DOCUMENT RESUME 

ED 378 364 CE 067 972 



AUTHOR 
TITLE 



INSTITUTION 

SPONS AGENCY 

PUB DATE 
CONTRACT 
NOTE 

AVAILABLE FROM 
PUB TYPE 



Stecher, Brian M. ; And Others 
Improving Perkins II Performance Measures and 
Standards. Lessons Learned from Early Impl emtmters in 
Four States. 

Natxonal Center for Research in Vocational Education, 
Berkeley, CA. 

Office of Vocational and Adult Education (ED) , 
Washington, DC. 
Jan 95 

V05U30003-94A; V051A30004-94A 
88p. 

NCRVE Materials Distribution Service, Horrabin Hall 
46, Western Illinois University, Macomb, IL 61455 
(order no. MDS-732: $6). 
Reports - Research/Technical (143) 



EDRS PRICE MF01/PC04 Plus Postage. 

DESCRIPTORS ^Accountability; Educational Legislation; Educational 

Policy; ^Federal Legislation; Federai State 
Relationship; Policy Formation; Postsecondary 
Education; "Program Improvement; *Public Policy; 
Secondary Education; "State Standards; Statewide 
Planning; "Vocational Education 

IDENTIFIERS "Carl D Perkins Voc and Appl Techn Educ Act 1990 



ABSTRACT 

A study examined implementation of the accountability 
and program provisions of the Carl D. Perkins Vocational and /pplied 
Technology Education Act of 1990 (Perkins II) in four unnamed states 
that were "early adopters" of the Perkins II measures and standards. 
In each state, sfaff in the state agencies administering secondary 
and postsecondary vocational education were interviewed along with 
administrators and instructors from secondary and postsecondary 
vocational institutions in two geographically separated regions. The 
interviews established that, although substantial progress has been 
made in implementing mer'.ures and standards, much work remains to be 
done to make the systems function as envisioned in Perkins II. Two 
sets of factors were identified as contributing to variations in 
slates' responses to performance measures and standards. The first 
set were within the sphere of influence of federal policy, whereas 
the s econd set f 1 owed from state/local contexts and are thus less 
directly subject to federal policy intervention. The following 
recommendations were directed toward federal policymakers: cls.rify 
the interrelationships among Perkins II mandates and coordination of 
Perkins II initiatives, create models for outcome-based program 
improvement , provide focused technical assistance regarding choices 
and resources, and address common measurement problems. (Appended are 
the state and local interview guides.) (MN) 



ft Vc Vc Vc Vc Vc Vc Vc Vc Vc ft Vc Vc ft Vc ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft 

* Reproductions supplied by EDRS are the best that can be made * 
x from the original document. * 

ft ft Vc Vc ft ft ft ft Vc Vc ft ft Vc ft V c ft ft ft ft ft ft ft ft ft ft ft Vc Vc Vc Vc Vc Vc Vc Vc Vc ft Vc Vc Vc Vc ft ft ft Vc Vc Vc Vc Vc ft Vc Vc ft Vc Vc Vc Vc Vc Vc Vc Vc Vc Vc Vc Vc Vc Vc Vc ft ft Vc Vc 



—% 

National Center for Research in 
Vocational Education 



University of California, Berkeley 



Improving Perkins II Performance 
Measures and Standards 

Lessons Learned from 
Early Implemented in Four States 



EDUCATIONAL RESOURCES INFORMATION 
/ CENTER (ERIC) 

>J This documenl has beon reproduced a- 

□ Minor changes have been made to 
imprc. > reproduction quality. 



document W °\ ° pinions * ,ated «n this 
aocument do not necessarily represent 
off-cial OERI position or poliby 



i 



Supported by 
The Office of Vocational and Adult Education, 
U.S. Department of Education 



2 

BEST COPY AVAILABLE 



This publication is available from the: 



National Center for Research in Vocational Education 
Materials Distribution Service 
Western Illinois UnWersity 
46 Horrabin Hall 
Macomb, IL 61455 



8(X)-637-7652 (Toll Free) 



Improving Perkins II Performance 
Measures and Standards 

Lessons Learned from 
Early Implemented in Four States 



Brian M. Stecher, Lawrence M. Hanser, 
and Bryan Hallmark, RAND 

Mikala L. Rahn, Karen Levesque, E. Gareth Hoachlander, 
David Emanuel, and Steven G. Klein, 
MPR Associates, Inc. 



National Center for Research in Vocational Education 
University of California, Berkeley 
2150 Shattuck Avenue, Suite 1250 
Berkeley, CA 94704 



Supported by 
The Office of Vocational and Adult Education, 
U.S. Department of Education 

January, 1995 MDS-732 
4 



FUNDING INFORMATION 



Project Title: 

Grant Number: 

Act Under Which 
Funds Administered: 



National Center for Research in Vocational Education 
V05 1A30004-93A/ V05 1 A30003-93A 

Carl D. Perkins Vocational Education Act 
P.L. 98-524 



Source of Grant: 



Grantee: 



Director: 

Percent of Total Grant 
Financed by Federal Money: 

Dollar Amount of 
Federal Funds for Grant: 

Disclaimer: 



Discrimination: 



Office of Vocational and Adult Education 
U.S. Department of Education 
Washington, D.C. 20202 

The Regents of the University of California 
c/o National Center for Research in Vocational 

Education 
2150 ShattuckAve., Suite 1250 
Berkeley, CA 94704 

Charles S. Benson 



100% 



$6,000,000 

This publication was prepared pursuant to a grant 
with the Office of Vocational and Adult Education, 
U.S. Department of Eduction. Grantees undertaking 
such projects under government sponsorship are en- 
couraged to express freely their judgment in profes- 
sional and technical matters. Points of view or opin- 
ions do not, therefore, necessarily represent official 
U.S. Department of Education position or policy. 

Title VI of the Civil Rights Act of 1964 states: "No 
person in the United States shall, on the ground of 
race, color, or national origin, be excluded from par- 
ticipation in, be denied the benefits of, or be subjected 
to discrimination under any program or activity re- 
ceiving federal financial assistance." Title DC of the 
Education Amendments of 1972 states: "No person in 
the United States shall, on the basis of sex, be ex- 
cluded from participation in, be denied the benefits 
of, or be subjected to discrimination under any educa- 
tion program or activity receiving federal financial as- 
sistance." Therefore, the National Center for Research 
in Vocational Education project, like every program or 
activity receiving financial assistance from the U.S. 
Department of Education, must be operated in com- 
pliance with these lavs. 



PREFACE 



The Carl D. Perkins Vocational and Applied Technology Education 
Act of 1990 (Perkins II) contained a number of significant changes 
from previous federal vocational education legislation. The act re- 
quired states to develop outcome-based accountability systems built 
around statewide performance measures and standards. States were 
given considerable flexibility in identifying outcomes f.o be mea- 
sured, selecting measurement tools, and setting standards. Local 
programs were given the principal responsibility for program im- 
provement, with states intervening only when local programs were 
unable to demonstrate significant progress. Measures and standards 
were supposed to be adopted by the fall of 1992 and used thereafter 
as accountability tools and guides to program improvement. 

This study is one of a series of investigations conducted by the 
National Center for Research in Vocational Education (NCRVE) relat- 
ing to vocational education accountability and the implementation 
of these measures and standards. The oresent research focuses on 
the effects of the performance measures and standards on state vo- 
cational education agencies, local programs, and the relationships 
between them. The study examines four states that were among the 
first to adopt measures and standards and implement data collection 
and reporting systems. Information gathered from these states gives 
some indication of the impact of the legislation among "early 
adopters" and the factors that affect implementation. The findings 
should be of interest to federal policymakers considering the reau- 
thorization of the legislation, as well as to state and local vocational 
educators looking for ways to improve the utility of their account- 
ability systems. 
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EXECUTIVE SUMMARY 



States have had three years to implement the accountability and 
program improvement provisions of the Carl D. Perkins Vocational 
and Applied Technology Education Act of 1990 (Perkins II). This 
study examined progress in implementing statewide systems of per- 
formance measures and standards, the effects of these systems on 
state agencies and local vocational programs, and the factors that in- 
fluenced state and local actions. 1 We offer recommendations for 
changes in federal policy to promote the goals of local accountability 
and program improvement embodied in the original legislation. 

PROCEDURES 

We selected four states that were "early adopters" of measures and 
standards for study. In each state, we inverviewed staff in the state 
agency (or agencies) that administered secondary and postsecondary 
vocational education and visited secondary and postsecondary vo- 
cational institutions in two geographically separated regions. Each 
institution was visited twice, once in the fall of 1993 and again in the 
spring of 1994. During the site visits, we interviewed administrators 
and instructors in each institution. 



] No fomial attempt was made to determine whether states were in compliance with 
Perkins II or to judge the quality of the measures and standards states had chosen to 
implement. 



9 

ERIC 



ix 

10 



x Improving Perkins II Performance Measures and Standards 



WHAT WE FOUND 

Substantial progress has been made in implementing measures and 
standards in the states that were visited, although much work re- 
mains to be done to make the systems function as envisioned in the 
law. By 1994, little attention had been paid to building state- or lo- 
cal-level capacity for translating the measures and standards data 
into actions at the local level for program improvement. These 
"leading edge" states were still largely engaged in developing and 
implementing their systems. 

Furthermore, wide variation was found in the states' approaches to 
the development and implementation of measures and standards. 
This variation was evident in almost every aspect of program imple- 
mentation, including how the process was managed, who partici- 
pated, and the level of resources devoted to it. These differences ap- 
peared to be jointly a function of the states' individuality and the 
flexibility inherent in Perkins II. 

WHAT EXPLAINS PROGRESS TO DATE? 

We identified several factors that contributed to the variation in state 
responses to performance measures and standards. Some of these 
explanatory factors are within the sphere of influence of federal pol- 
icy, including: 

• Flexibility provided to the states in the law itself. The flexibility 
of the Perkins II mandate for measures and standards had mixed 
effects. It permitted states to create systems that were respon- 
sive to local conditions, but it also increased the influence of 
state and local contextual factors on implementation, which 
slowed and limited the process in some cases. 

• Unclear integration and coordination of Perkins II provisions. 
In the states we visited, the Perkins II priorities — measures and 
standards, integration, tech-prep education, 2 and service to 



2 "The term 'tech-prep education program' means a combined secondary and post- 
secondary program which leads to an associate degree or 2 -year certificate; provides 
technical preparation in at least 1 field of engineering technology, applied science, 
mechanical, industrial, or practical art or trade, or agriculture, health, or business; 
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special populations — were treated as separate activities. They 
were not seen as a coordinated system at either the state or local 
level and performance measures and standards were not being 
used comprehensively to evaluate the other Perkins initiatives. 

• Lack of models or incenti r es for state and local implementation. 
Perkins II contains an explicit framework for structuring systems 
and federal agency checks for compliance at the adoption stage, 
but there are neither models nor incentives for ensuring that 
performance measures and standards are used to improve 
programs. 

• Limited expertise and resources at the state level. Perkins II 
created new responsibilities for state staff while reducing the set- 
aside for state administration. This presented a dilemma for 
some states that lacked both the expertise and the resources to 
address these new demands. 

• Mandated measurement of learning outcomes. The scarcity of 
valid assessment tools for measuring selected learning outcomes 
led states to adopt alternatives that were less than optimal. 
States are still struggling with how to measure important student 
outcomes, such as academic skill gains at the postsecondary 
level. 

The second set of factors flows from state and local context, and 
hence, these factors are less directly subject to federal policy inter- 
vention. These included the choice betweem a centralized and a lo- 
cal implementation strategy, the existence of statewide data systems, 
the availability of state-level assessment tools, the nature of ongoing 
educational accountability and program review mechanisms, histor- 
ical relationships among state and local educational and employ- 
ment agencie attempts to reduce the burden on local administra- 
tors, and the influence of key administrators. 



builds student competence in mathematics, science, and communications (including 
through applied academics) through a sequential course of study; and leads to place- 
ment in employment." (The Carl D. Perkins Vocational and Applied Technology 
Education Act Amendments of 1990. P.L. 101-392; Part E, Sec. 347.) 
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Finally, there are issues on the horizon that will affect the implemen- 
tation of measures and standards and should be considered in plan- 
ning the reauthorization of Perkins: 

• Skill standards and curriculum standards. The Departments of 
Education and Labor are funding the development of industry 
skill standards in more than 20 occupational areas and curricu- 
lum standards in six educational fields. There should be some 
coordination between these efforts and the systems of measures 
and standards developed under Perkins II. 

• Data quality. States have not yet addressed the questions of reli- 
ability and validity of the measurement tools they have selected. 
Data quality questions will become more important as states 
begin the accountability and program improvement cycle, which 
will add high stakes to the performance measures and standards. 

• Integration of academic and vocational education. The inte- 
gration of academic and vocational education raises a host of 
problems for defining, measuring, and attributing outcomes and 
therefore threatens the validity of existing systems of measures 
and standards. 

• Consistency in federal vocational education policy. Some state 
vocational educators believe new laws will supplant many of the 
initiatives contained in Perkins II; as a result, they make only 
halfhearted efforts at implementation. In this way, the volatility 
of federal vocational education policy discourages rapid and 
effective response to federal initiatives. 

RECOMMENDATIONS 

Federal policymakers may take several actions to enhance the future 
success of performance-based accountability in vocational educa- 
tion and promote the goals of Perkins II: 

• Clarify the interrelationships among Perkins II mandates and 
the coordination of Perkins II initiatives. Perkins II contains 
four major priorities: integrating vocational and academic edu- 
cation, providing service to special populations, creating tech- 
prep education programs, and establishing systems of measures 
and standards. Two of these are primarily curricular changes; 
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one relates to the selection of students; and one relates to ac- 
countability and program improvement. In theory, these efforts 
should complement one another, and states' efforts to address 
the priorities should be coordinated. In fact, none of the four 
states coordinated their efforts to address these mandates, and 
there was wide variation in the relative priority assigned to these 
four critical initiatives. Policymakers should clarify the inter- 
relationships among systems of measures and standards, the in- 
tegration of academic and vocational education, tech-prep edu- 
cation programs, and service to special populations and offer 
additional guidance about coordinating states 1 efforts in these 
areas. 

Create models for outcome-based program improvement In 
1994, most state action was still driven by the mandate to de- 
velop a structure for accountability, i.e., the system of measures 
and standards. Little had been done to use that structure to 
make programs better. State and local agencies need assistance 
in translating outcome deficiencies into action plans. One ap- 
proach might be to require states to develop program improve- 
ment models that illustrate how outcome-based information can 
be used for local program reform. The alternative we suggest 
would be to commission an agency other than the states them- 
selves tc collect examples of effective outcome-based local im- 
provement practices and disseminate them widely for states and 
local programs to use. 

Provide focused technical assistance regarding choices and re- 
sources. The "flexible mandates" of Perkins II place greater de- 
mands on state agencies yet restrict the use of funds for state- 
level services. These fiscal restrictions come at a time when 
many state administrative budgets are also being reduced. 
Under these circumstances, federal actions that help states re- 
spond to their choices and make hotter use of resources might 
significantly improve implementation of the act. We also suggest 
increasing the funds available for state administration during the 
start-up phase, so states can meet initial demands and develop 
some of the expertise they will need to operate a reformed voca- 
tional education system. 

Address common measurement problems. The technology to 
measure learning and occupational performance gains in reli- 
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able, valid, and efficient ways is not widely available. Most states 
are not equipped with either the resources or expertise to de- 
velop tools for measuring learning and occupational outcomes, 
and it is unfair to require them to accomplish this difficult task. 
The federal government needs to assume leadership in address- 
ing these problems, since they are best solved nationally and are 
largely the result of the provisions of Perkins II. 

Incorporating these changes into the reauthorization of federal vo- 
cational education legislation will increase the efficacy of statewide 
systems of measures and standards. Legislators also should antici- 
pate difficulties and conflicts that may be created by pending reforms 
of education, training, and workforce preparedness programs and 
should work to coordinate the accountability requirements of all 
these initiatives. 
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Chapter One 



INTRODUCTION 



It has been three years since states began to implement the account- 
ability and program improvement provisions of the Carl D. Perkins 
Vocational and Applied Technology Education Act of 1990 (Perkins 
II), and policymakers are asking whethei these provisions are work- 
ing. We know, from previous National Center for Research in 
Vocational Education (NCRVE) research, that states adopted systems 
of measures and standards as prescribed by federal law (Rahn et al., 
1992), but we do not know whether, or how well, these systems were 
implemented, or whether they are leading to program improvement. 
These two questions are particularly relevant at present, because it is 
time for the reauthorization of federal vocational education legisla- 
tion. Policymakers are looking for information about the success of 
these locally focused, outcome-based systems of program improve- 
ment and about ways to improve the federal rules that govern them. 

This study examines the implementation of Perkins II performance 
measures and standards to date and, based on this interim review, 
recommends actions the federal government can take to improve 
these systems. Ultimately, Perkins II should be judged in terms of 
outcomes: Has the legislatfon established and/or strengthened local 
systems of outcome-basec 1 program review and improvement, and 
have these systems led to improved workforce outcomes at the local 
level? 

However, it may be too soon to make these judgments. States are 
only now entering the first program improvement cycle, and 
widespread data on outcomes are at least one to two years away. 
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Consequently, we framed our research in terms of two more imme- 
diate questions: 

• Are states implementing their systems of measures and stan- 
dards as envisioned in the federal legislation? 

• What factors have affected their progress, and can progress be 
enhanced through federal action? 

Before presenting our results, we provide some background on the 
requirements of Perkins II and on relevant NCRVE research, 

LEGISLATIVE REQUIREMENTS 

Perkins II called for tho creation of performance-based state ac- 
countability systems for improving vocational programs. One of the 
most dramatic changes in Perkins II was the addition of outcome- 
based performance measures and standards in these accountability 
systems, particularly the inclusion of learning outcomes that were 
seldom specified in previous evaluations of vocational programs. 

Perkins II required each state to "develop and implement a statewide 
system of core standards and measures of performance" by 
September 1992 (Public Law 101-392, Section 1 15(a)). The mandate 
has three key distinguishing features: (1) i iphasizes the use of 
outcomes to monitor program success; (2) n gives states consider- 
able flexibility in creating systems; and (3) it assigns primary respon- 
sibility for program improvement to local programs. 

At minimum, each state was required to include in its system at least 
two sets of measures. The first must be a measure of "learning and 
competency gains, including student progress in achieving basic and 
more advanced academic skills." The other set must include any one 
of the following four measures: (1) competency attainment; (2) job 
or work skill attainment; (3) retention or completion in school; or (4) 
placement in further education, the military, or employment. 

Although Perkins II prescribes the basic guidelines that states must 
follow in designing accountability systems, it also provides each state 
considerable discretion in developing and implementing a system 
that fits its individual situation and needs. For example, some states 
have taken a centralized approach to designin 6 their measures, pre- 
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scribing specific assessment instruments or data collection proce- 
dures to be used by local recipients, while others have taken a decen- 
tralized approach, charging local recipients to develop their own in- 
struments or procedures. At the time of our research, states had not 
all reached the same stage of implementation of their selected sys- 
tems of measures and standards. 

PRIOR NCRVE RESEARCH 

Since Perkins II was enacted, NCRVE has been actively involved in 
providing technical assistance to state-level administrators to assist 
them in developing and implementing systems of performance 
measures and standards. NCRVE researchers wrote a guidebook, 
Accountability for Vocational Education: A Practitioner's Guide 
(Hoachlander, Levesque, and Rahn, 1992), which was retributed at 
regional workshops and has been widely used by state-level adminis- 
trators to explore key issues in designing an accountability system. 

NCRVE researchers also examined several local vocational education 
accountability systems to gather information that might help im- 
prove the implementation of Perkins II. Beyond Vocational Edu- 
cation Standards and Measures: Strengthening Local Accountability 
Systems for Program Improvement (Stecher and Hanser, 1993) 
described local accountability systems that already existed for many 
vocational programs and showed how those systems could be used 
for program improvement. The authors suggested ways that states 
and local programs might improve the functioning of local 
accountability systems. 

Continuing interest in obtaining information about state responses 
to the performance r.oasures and standards req irements led 
NCRVE to compile State Systems for Accountability in Vocational 
Education (Rahn, Hoachlander, and Levesque, 1992). This report 
contains a summary of the performance measures and standards 
implemented by states in the fall of 1992 and provides examples of 
clearly defined measures and standards. The appendix briefly de- 
scribes each state's system of performance measures and standards 
at both the secondaiy and postsecondary levels. The report is based 
not only on information gathered through workshops and telephone 
interviews but also on analysis of documentation submitted to 
NCRVE. 



19 



4 Improving Perkins II Performance Measures and Standards 



OVERVIEW 

This study examines how some spates are progressing in achieving 
the accountability goals outlined in Perkins II. It focuses on the pro- 
cess of implementing performance measures and standards for vo- 
cational education at the state and local levels. It is too soon to tell 
how the implementation will work out in these states, let alone to 
know whether the systems will have the desired effects of institu- 
tionalizing local outcome-based program improvement systems. 
Nevertheless, results to date can be used to judge the degree to which 
states are succeeding in achieving the vision embodied in Perkins II, 
what factors affected their progress and how federal legislation 
might promote greater success in the future. 

In this study, no attempt was made to judge whether the states were 
in compliance with the legislation. The Office of Vocational and 
Adult Education (OVAE) is the agency responsible for determining 
compliance, not NCRVE. Moreover, no formal assessment was made 
of the quality of the measures and standards themselves in these 
states. Perkins II requires the Secretary of Education to submit a re- 
port to Congress that assesses "the validity, prcdicriveness, and reli- 
ability" of standards and measures. The secretary's report may in- 
clude some measures of quality. 

Our report is organized as follows. Chapter Two briefly describes the 
research procedures. Chapter Three presents findings regarding 
state implementation and includes brief summaries of the evolution 
of performance measures and standards at the secondary and post- 
secondary levels in each state. Chapter Four discusses the factors 
that affected states' actions, including both conditions that may be 
subject to federal actions and local and state contextual factors that 
are within the domain of state policy. The discussion also identifies 
emerging conditions that are likely to affect the implementation of 
Perkins II in the near future. The final chapter recommends actions 
to enhance the implementation of these systems. 
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PROCEDURES 



SAMPLING 

A small purposive sample of four states was chosen to develop a 
deeper understanding of the implementation of Perkins II measures 
and standards from states that were farthest along in the process as 
of January 1993 (Rahn, Hoachlander, and Levesque, 1992). This 
sampling strategy has two advantages. First, it permits us to extrapo- 
late and draw inferences from states that have completed more of the 
required steps in establishing outcome-based accountability sys- 
tems. That is, because these states are further along, they have 
greater experience with more aspects of the system and its imple- 
mentation. Second, and as a direct result of their experiences, our 
strategy enables us to identify models that might inform other states 
and improve their systems. For example, state and local strategies 
that worked in our sample of states might be adopted by states 
struggling with implementation. 

T he major disadvantage of this strategy is that the four states are not 
necessarily representative of the country as a whole. Conditions in 
these four states might not be duplicated in others, and actions that 
worked here might be less successful elsewhere. As a result, the 
reader should be cautious about extrapolating these findings or in- 
terpreting them as repre& tative of the country as a whole. While 

the findings from the four states in our sample may not be generaliz- 
able, we believe our recommendations, if implemented, would 
benefit many states in their implementations of Perkins II. 
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DATA COLLECTION 

In each state, we visited the state department of education and se- 
lected secondary and postsecondary institutions in two different re- 
gions. The secondary and postsecondary institutions we visited in 
these regions were chosen by the state, in consultation with us. We 
asked the state departments of education to choose institutions typi- 
cal for their state. We also asked for variety in terms of economic 
conditions and encouraged the states to choose one urban and cne 
rural region. 

A team of two researchers spent a total of approximately one and 
one-half weeks in each state over the course of two visits (Fall 1993, 
Spring 1994). At the school districts, we spoke with administrators 
and instructors and asked about the implementation of Perkins II 
and its effects. We developed a structured interview guide with tai- 
lored questions for chief administrators (including district super- 
intendents, college chancellors, and college presidents), principals, 
directors of vocational education (including college vocational ad- 
ministrators), and instructors. After determining his or her famil- 
iarity with the state-adopted system of measures and standards, we 
questioned each interviewee on the following topics: 

• The importance of measures and standards vis-k-vis other voca- 
tional education initiatives (For example, which vocational edu- 
cation reforms or initiatives received the most emphasis in your 
institution in the past two years?) 

• The integration of vocational education accountability with 
other educational reforms (For example, is accountability an im- 
portant aspe( t of general education reform? If so, how have vo- 
cational education measures and standards affected this process 
or been affected by it?) 

• Implementation of measures and standards (For example, what 
steps have been taken to implement performance measures and 
standards from the time they were adopted until the present?) 

• Effects of measures and standards (For example, what changes 
have occurred in program evaluation procedures as a result of 
measures and standards?) 
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• Technical assistance anu support for measures and standards 
(For example, what technical assistance has been provided by 
the state? By the district?). 

Copies of our interview protocols can be found in the appendixes. 
DATA ANALYSIS 

We compiled extensive notes in the course of our interviews, which 
we then edited, summarized, and distributed to all members of the 
research team. Impressions gained in initial visits to the states were 
tested during subsequent visits, and respondent comments were 
compared across institutions. The findings documented in this re- 
port represent the results of many discussions and formal meetings 
during which we attempted to synthesize what we had collectively 
observed over the span of several weeks in the field. 
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WHAT WE FOUND 



We begin this chapter with brief summaries of the status and most 
salient elements of each state's implementation of Perkins II. It is 
impossible to convey the full complexity of each state's system in 
such short summaries; however, we have tried to provide enough 
information to help the reader distinguish among the states and to 
introduce distinct characteristics of each state's approach. 

Following that, we summarize the overall progress these states made 
in implementing statewide systems of performance measures and 
standards and compare some of their choices. We also found con- 
siderable variation in the way the four states responded to the 
Perkins II mandates — they followed different procedures, selected 
different measures and standards, and implemented their systems in 
different ways — and this variation is discussed next. The section 
concludes with observations about how much remains to be done, 
even in these four selected states. 

STATE SUMMARIES 

We refer to the four states by the pseudonyms Alcorn, Columbia, 
Erie, and Piedmont. This convention serves both to protect the 
anonymity of our sources and to preclude interpretations based on 
existing attitudes toward particular jurisdictions. Most states created 
two distinct systems — one at the secondary level and another at the 
postsecondary level — so summaries differentiate between secondary 
and postsecondary vocational institutions. 
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Alcorn 

Alcorn is a predominantly urban state with a large agricultural sector. 
It has a major metropolitan area that dominates the economy of the 
state. At the secondary level, there is an extensive accountability sys- 
tem that includes academic skill exams, a performance-based quality 
review process on the academic side, and schoolwide report cards. 
The state hopes to incorporate performance measures and standards 
into this "academic" accountability system. 

Alcorn is in the process of creating a workforce preparation system 
that will include ten measures and standards at both the secondary 
and postsecondary levels. This system includes piloting of workplace 
readiness instruments and the development of industry skill stan- 
dards and certification. The primary goal of performance measures 
and standards in Alcorn is to produce high-quality data that can be 
used to compare the progress of programs across the state. The state 
made early strides in meeting this goal because of its existing com- 
puting capacity and centralized approach. There was evidence of 
interagency coordination to obtain access to new outcomes data and 
devotion of staff time to develop a strong data system. However, the 
complexity and centralized nature of the state's data system have re- 
sulted in the delayed dissemination of data to districts. Litde atten- 
tion has been paid to local implementation, particularly how instruc- 
tors might use the data to improve programs. 

At the postsecondary level, the regional accreditation body leads the 
main accountability initiative. This accreditation effort is moving 
toward requiring postsecondary institutions to include outcome- 
based measures. Several other initiatives are pushing community 
colleges in this direction. Alcorn proposes to establish an "insti- 
tutional guarantee" in which postsecondary institutions guarantee 
employers that students have acquired certain occupational skills. 
The state would like to develop a system that will link accreditation, 
the state-led program review, and performance measures and stan- 
dards while also using work-readiness assessment instruments and 
industry skill standards. 

As with the secondary level, however, there has been little evidence 
that Perkins II performance data have been used by deans and in- 
structors for program improvement purposes. Both deans and re- 
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gional directors reported that the scope of the data they received was 
overwhelming and that a strategic targeting of the most relevant data 
would be necessary to make it useful for improving programs. 

Columbia 

Columbia is a historically agricultural state with a growing industrial 
base and expanding school system. Educational accountability has 
been a major concern of the state legislature for the past decade. 
Annual school report cards that focus on outcomes are issued for 
each district and postsecondary institution. Poor performance af- 
fects a small part of the pool of funds available for teacher salaries. 
This accountability system put the state a step ahead in terms of im- 
plementing Perkins II measures and standards. 

At the secondary level Columbia has a strong central educational 
bureaucracy. Schools report data to the state at the student level, 
which the state analyzes and uses to produce required reports. After 
reviewing existing data-collection efforts, Columbia adopted eight 
centralized measures; data for three of the measures were already 
being collected. The vocational education coordinator is an effective 
leader who involved staff across the state and created a vision for 
program improvement. Under her direction, the state moved ag- 
gressively to expand a computerized test bank and instructional 
management system to cover all its vocational courses. This system 
was developed in anticipation of changes in federal vocational edu- 
cation, and funds from Perkins II were used to transform it into a 
testing system to provide required occupational measures. 

Postsecondary education in Columbia is more decentralized, with 
institutions retaining considerable autonomy. For example, al- 
though they have to produce report cards, each institution collects 
and analyzes its own data. The vocational education coordinator re- 
spects this autonomy and tries to shield the institutions by reducing 
the "burden" of federal requirements. The state relied on existing 
measures to form the backbone of its postseconclmy Perkins II 
statewide system. 
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Erie 

Erie is primarily a rural state with a growing student and minority 
population. Elementary and secondary enrollments increased over 
10 percent from 1980 to 1992, and postsecondary enrollments grew 
over 25 percent from 1987 to 1991. Accountability is important in 
Erie— recent state legislation requires standardized assessment in 
several grades. At the same time, school districts have a long history 
of autonomy and retain a great deal of independence. Coordinating 
state and federal legislation, in conjunction with wide local discre- 
tion, has been difficult for the state in its efforts to implement the 
Perkins II measures and standards. Additionally, unclear communi- 
cation at the local level has confused local agencies about perfor- 
mance measures and standards. 

Having a single state department that is responsible for both sec- 
ondary and postsecondary occupational education eased the process 
of selecting measures and standards for Erie. The state adopted a 
total of nine measures— six based on data that the state was already 
collecting. The other three were defined conceptually by the state 
but were passed to the local agencies to operationalize and imple- 
ment. These three are to be measures of academic and occupational 
achievement — perhaps the most difficult to operationalize and im- 
plement with local resources. 

Staffing changes at the state level also resulted in a short hiatus in 
contact between the state and local agencies. As a result, the local 
secondary and postsecondary agencies we interviewed have largely 
ignored the six measures and standards the state was already collect- 
ing and have been perplexed with the three that they were given to 
operationalize. The state and local agencies could be best described 
as "regrouping" in their efforts to implement the measures and stan- 
dards. The state is reconsidering the set of measures and standards it 
selected, and local secondary and postsecondary agencies are begin- 
ning to struggle with operationalizing and implementing the three 
measures and standards that are their responsibility. 

Piedmont 

Piedmont is a predominantly urban state with a growing and 
increasingly diverse population. Its major metropolitan area has 
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grown dramatically in the last decade and has become the regional 
hub for business and financial services. Perkins performance 
measures and standards represent the main accountability initiative 
for vocational education in Piedmont at the secondary level. 
Traditionally, local agencies have enjoyed autonomy, with the state's 
role primarily limited to compliance monitoring. However, with the 
implementation of performance measures and standards, the state's 
role has dramatically shifted from monitoring compliance to 
providing technical assistance to foster program improvement. The 
state adopted four outcome areas and standards; each district chose 
or developed its own assessment instruments to evaluate students in 
these areas. Vocational teachers have played an active role in 
developing pre- and post-tests to measure related academic gain and 
competency attainment. This has encouraged vocational teachers to 
think about the academic skills related to the occupational 
competencies they teach. The state developed an extensive technical 
assistance program for local districts, including a detailed guide to 
collecting Perkins-related data and a series of regional workshops for 
vocational education directors. 

At the postsecondary level, Piedmont is even more decentralized, 
with each ; nstitution having considerable autonomy. The vocational 
education coordinator respects the autonomy of each institution and 
in genera] has tried to minimize the data burden on vocational deans 
and instructors. At the postsecondary level Piedmont adopted a 
scaled-down version of the secondary performance measures and 
standards — most of the system to be phased in slowly. The main ac- 
countability initiative at this level has not come from the state but 
from the regional accreditation body. To minimize the data-gather- 
ing burden, the state originally decided to use the same cohort of 
students that the federal student right-to-know legislation requires. 1 
However, this cohort proved to be too limited ana not useful for the 
purpose of improving programs. The vocational deans plan to ex- 
pand the cohort this year. Because of the limited cohort and the im- 
portance placed on the accreditation system, there has been little ev- 



^The Student Right to-Know and Campus Security Act, P.L. 101-542, November 8, 
1990, requires postsecondary institutions to collect and publish data on the comple- 
tion or graduation rate of certificate or degree-seeking fuU-time students. Regulations 
implementing this act were still pending in 1994. 
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idence that performance measures and standards have been widely 
adopted and used for program improvement purposes. There is a 
perception among some faculty members that performance mea- 
sures and standards represent another superfluous federal "add-on" 
program. 

IMPLEMENTATION PROGRESS TO DATE 

We selected these four states for this study, because they were among 
the "early adopters" of measures and standards and had made con- 
siderable progress in implementing their statewide systems at the 
time of the study. The committees of practitioners in all four states 
had formally adopted the measures and standards; to the best of our 
knowledge, all four states were in compliance with the Perkins II 
regulations. On average, these states had adopted seven outcomes at 
the secondary level and nine outcomes at the postsecondary level. 
Table 3.1 shows the breadth of the measures adopted at the sec- 
ondary level. All four states adopted measures of academic skills, 
specific occupational competencies, and labor-market outcomes. 
Most also had measures of general job or work skills and of program 
retention and/or completion. The pattern was similar at the sec- 
ondary level. Table 3.2 shows that three of the four states adopted 
postsecondary measures in the majority of these major outcome ar- 
eas. 

Some of the states were phasing in one or more elements of their sys- 
tems over a one- or two-year period, but the vast majority of the 
components were fully operational in the 1993-94 school year. Data 
were collected during that year for most measures, although analysis 
and reporting of these data will not occur until the 1994-95 school 
year in many cases. 

In addition to the measures described in Tables 3.1 and 3.2, all four 
states either had or were actively developing the necessary informa- 
tion infrastructure to support standards-based accountability. 
Columbia and Erie have computerized student recordkeeping sys- 
tems in place that could be used for analyzing and reporting progress 
toward state standards and functioning paper-based data systems for 
collecting most of the remainder of the statewide measures. Alcorn 
was expanding its existing centralized data-collection and analysis 
system to accommodate newly adopted measures, while Piedmont 



29 



What We Found 



15 



CO 



« 

a 
o 



g 

S 

s 

ir 

G 
O 

u 

0J 
C/5 



re o 



CO ■ 

U US 

° to-* 
^1 cd 

° TJ '5 
co CD G 



c 

\J co 



1 « CO 



cd 
re 

c 

c 2 
5 £ 



(LI 

CO CO 

8* 
*- u 

S E 

E-S 

C rf 
« «J 
< o 



- CO CO 

- ° y 

: 0) cd 

- *cB o 

C co u 



< 



c 

US 

I ° 
8 gi£ 

g o § 

.is"? 



_ I— 

5-g.S § 

S g-3 9 

iilll 




c 

« S 
c .2 

'5s « 

00 fx 

c « y 

§ S H 

•i2 | 

*^ t/5 5 

m O 



C la 



c o u 
cd *G co 

E^-S 
•I 8 = 

fS ~ cu 
< O CO 



"CO .2 
C U 

o c 

CP 

8.8 S 

on 



o 

CO co 

w a> 

c ~ 

0) c 

E o 

.b co 

CO C 

G « 

< ao 



2 -a 
S » c 

g a> o 

O J2 3 
O r 



■o 

2 g 
g E 

•Q CO 

co O 
O G« 

CU SO QA 

60S c 
CO .G 



.£ o 
2 * 



< o 



•S 2 

O co 



u. O 



C CO 

CD CL 3 

h 5 CO 

QJ O 



O CD 



CD 



CO CD 

G g\S- 

Sac B 

CD *~ 



.EH 

CU CD O O 



CD 

cd 



DO 



£ I - 5 

O J2 CD CO 
U CD JZ »- 
i- ■fc- 

- < J; u 
: 2 ° 



CD ' 

oo«o c 

CO CD -G 

e st 1 
§° O 

CD £ CD TJ 
CL CD <G CD 



O 

«5 



C . CU c 

re c 



o 

U "~ ku 

o c 
d 

nc c 

O C CD 

IP 

C G CD 

C CJ > 



C CD 

3 *a 



o 
c 



CD 



co o 



o 

-C 



c 

C CD 
£ E 
£ • O 

?S C ° 
•O CD G 

CD E o 
i £ cu c 

! CO CD «5 

! Si-ScS 

3 S3 £ 
re T3 <« 




16 



Improving Perkins II Performance Measures and Standards 



CO 

3 

H 



cd 

CO 

1m 

3 
O 



s 



a 
o 

u 
cy 



O 



J22 £ 

CX CD 
P T3 

C 3 
U to 



u 
o t 

p 

o £ 

to 
C 0) 
CD *^ 
C "O 

P S 
CO cd 

w cd 



<«>i cu 

Si 



.2 co 
cx a 

P 3 

5 <-> 

O u 

U o 



o £ 



< 



< 



OS - 3 

• ■ C «j 

O CO 

w CD 

c p 



to E 



<C CO 



•I " 

cd cd 



73 « 

C y 

o c 

Cw rv 

3 p 

u O 

O O 



to 
CD CD 

c <H 3 
g ? S 

< O <J 



2 c 

— c 0) 
.sou 
cd °3 
« u £ 
01*3.10 

£ C ^ 

o c »- 
►3 u c 



c i S .£ o 
<■> >. »- '3 
<s O o id 

CO &2 3 
CO S C Qj w 

§12Ji 



T3 
CD CD 

2.2 r- P. 

cd P - «• 



~ .2 T3 "O 

cd £ co cd .S 
o u *- ~ '3 



"3 c 

CD CO 
C -o 

CD flj to 

003 CD 

^- (0 W 

p *- o 

° P ° 

,R CD ^ 

D cd E Z 



CD 



X) 

< 



3 

cr 

CD 



CD .3 



30.W 
O 

u o c 

22 ?> E 

P H. c 

p cr cd 

O CD CD 



O 
P 

.2 

a; co 
"3. $ 

68 



' X) 
2 Era 

CO M I" 

8SJ 

CD -= CO 

eg .22 
«2a 

< O co 



p y 

o 3 

•£3 CD 

CO fl) 

*3 3 C 

O C_> C 

cd y O 

a. ° u 

CO 



CD O 

.So 

*-* to 



T3 CO 

2 ^ 

CO O 

C D. 
CD DO 
00 3 
CO *3 



S a3 c 

_0JC c 

Cw-3 "3 

ECU cc 

C J5 3 

U CD "3 

im fc- CD 

O C i- 

CD '*** Ct. 

00*0 -3 

3 3 
O CU # 3 C 

2> E *H g 

CD C 3 



3 

o 



a. 



00 



73 o 

CD ? 

3 i- 

CD O 

O 



si 

c £ 
S2 o 

CD U 



X3 
3 

CD 
to 
O 



E 
o 



o 

3 



CD »- 

co x: 



lERiC 



What We Found 17 



was devoting considerable resources to developing local capacity for 
data collection and analysis consistent with its decentralized ap- 
proach to measures and standards. 

All four states made efforts to coordinate the implementation of their 
systems of measures and standards with other state and federal oc- 
cupational training and economic development programs, as re- 
quired by the Peikins II guidelines. Unfortunately, the states found 
very little common ground between the measures and standards de- 
veloped by vocational educators in response to Perkins II and the 
data systems in use by other agencies. Although all four states at- 
tempted to coordinate data elements among their diverse economic 
development efforts, none had great success in this effort. Many 
states were working toward using unemployment insurance wage 
records as placement data across agencies. 

Finally, all four states had begun to prepare local educational agen- 
cies for their roles in the standards-based accountability system. A 
few of the local school or college staff people we interviewed were in- 
volved directly in the selection or adoption of measures or standards; 
they were familiar with the requirements from the outset. The re- 
mainder of the local staff had learned about the new requirements 
subsequently. Almost everyone we spoke to was aware of the mea- 
sures and standards in general terms if not in specific detail. Three of 
the four states informed local programs about the data demands and 
the accountability standards in fairly traditional ways, assigning dis- 
semination and training responsibility to existing state program or 
area staff, distributing written information, presenting regional 
workshops, etc. Efforts in the fourth state went well beyond those in 
the others, perhaps because local programs were responsible for se- 
lecting the measurement tools to be used to assess outcomes. In this 
state, the number of workshops and the amount of contact with local 
program staff were considerably greater than in the others. 

VARIATIONS IN IMPLEMENTATION 

One of the most striking findings from our interviews and observa- 
tions was the large degree of variation in the way each of the four 
states implemented Perkins II requirements. States' approaches to 
Perkins II measures and standards differed in almost every respect, 
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including who participated in the process, how the process was 
managed, what resources were devoted to it, how the regulations 
were interpreted, and how the system was integrated with existing 
state and local initiatives. 

The inherent "flexible" nature of the directive in Perkins II to develop 
and implement a system of performance measures and standards 
encouraged states to customize their systems to meet local needs. 
Moreover, states interpreted the law in different ways, which led to 
further variation in implementation. For example, states had differ- 
ent interpretations of the requirement to measure learning and 
competency gains. Some felt that the measurement of "gain" re- 
quires pre- and posttest scores; others chose to use course comple- 
tion as a proxy for improvement. In the absence of strong federal 
control over interpretation, states acted in the spirit of "flexibility," 
creating their own interpretations. In the next chapter, we attempt 
to describe some of the factors that affected states' interpretations 
and responses to Perkins II. 

Implementation at the postsecondary level is moving more slowly 
and is more varied than at the secondary level. This difference in 
implementation is due largely to the historical autonomy granted to 
postsecondary institutions. The movement toward educational ac- 
countability has only recently come to the postsecondary level, and 
there it faces unique challenges. For example, except for the pur- 
poses of initial placement, postsecondary institutions do not use as- 
sessment systems to test the achievement of students. Lacking basic 
achievement measures, it is almost impossible to measure learning 
"gain." A second factor in slowing the implementation of Perkins II 
at the postsecondary level is the diversity of student purposes that 
postsecondary institutions must serve. It is difficult to establish a 
system to track the progress of community college students with dif- 
ferent purposes for attending schools. An "entering" class will in- 
clude eighteen year olds out of high school, adult workers who need 
one course to upgrade their skills, and unemployed people who need 
retraining. 

ROOM FOR IMPROVEMENT 

Although these four states were among the leaders in implementing 
Perkins II measures and standards, their systems were only partially 
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complete at the time of this study. No state had yet completed a full 
cycle of data collection, analysis, and reporting on all adopted mea- 
sures, and no local program had yet examined its own performance 
in light of statewide standards as a guide to program improvement. 

Additionally, many states still had proposed measurement tools un- 
der development. Columbia proposed to measure literacy in terms 
of pretest to posttest gains, but had not yet taken any steps to de- 
velop a literacy assessment instrument. Common unresolved testing 
or assessment problems included the measurement of academic and 
occupational skills and difficulty setting up systems to monitor the 
performance of special populations. 

Most states' efforts were focused on creating statewide systems that 
would be in compliance with the federal regulations; much less 
thought Had been given to the use of the data the system produced 
for program improvement. Moreover, both state and local agencies 
had done little to translate information on program status into ac- 
tions for program improvement. We saw only one or two instances 
in which thought had been given to converting standards-based re- 
ports into action plans. 

Finally, with the exception of one highly decentralized state, most 
state departments of education were only beginning to provide ex- 
tensive information and training to local program staff. By the spring 
of 1994, states had disseminated to local districts the new rules and 
data requirements introduced by the statewide system. States were 
just beginning to confront the need for more local staff development, 
for greater contact with programs as they prepared new annual ap- 
plications, and for linking accountability with program improve- 
ment. 

The states in our study have made a good start in implementing their 
systems of measures and standards, as envisioned in Perkins II. 
However, conditions at all levels of government have resulted in 
some states making greater progress than others. In the next chap- 
ter, we attempt to explain why variations in progress have occurred. 
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WHAT EXPLAINS PROGRESS TO DATE? 



We identified a number of factors that contributed to the observed 
similarities and differences in states' implementation of Perkins II 
systems of measures and standards. Some of these explanatory fac- 
tors are within the sphere of influence of federal policy; others are 
within the domains of states and local agencies. In Chapter Five, we 
will draw upon the federal factors we discuss in this chapter to make 
recommendations for changes in federal policy to improve the im- 
plementation of Perkins II. 

FACTORS SUBJECT TO FEDERAL INFLUENCE 

The first set of factors we identified includes elements of the 1990 
Perkins legislation and other conditions that may be directly influ- 
enced by federal policy. These factors include the following: 

• Flexibility provided to the states in the law itself 

• Unclear integration and coordination of Perkins II provisions 

• Lack of models or incentives for state and local implementation 

• Limited expertise and resources at the state level 

• Mandated measurement of learning outcomes- 
Modifying these features through legislative or administrative means 
may enhance future implementation of Perkins II measures and 
standards in the states. We discuss each of these factors in greater 
detail below. 
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Flexibility Provided to the States in the Law 

The framers of the legislation sought to impose a common outcome- 
oriented program-improvement framework. They also wanted to 
enact policy that was sensitive to state differences and that permitted 
local adaptation, which research suggests should foster implementa- 
tion (Berman and McLaughlin, 1978). As a result, Perkins II gave 
states considerable latitude in choosing measures, setting standards, 
deciding how much responsibility to delegate to local districts and 
institutions, and designing program-improvement interventions. 
This flexibility permitted states to create systems that weie respon- 
sive to local conditions but also increased the influence of contextual 
factors, which had both positive and negative effects. 

On the positive side, the flexible mandate engaged states actively in 
developing rheir systems of measures and standards, gave states the 
opportunity to develop a sense of ownership, and enabled states to 
adapt the accountability system to their existing program review and 
monitoring efforts, potentially reducing costly duplication of effort. 
Furthermore, Perkins II gave states complete control over the nature 
of state assistance to failing programs, so these efforts could be 
aligned with ongoing program review and improvement activities. 
On the negative side, flexibility has increased the influence of local 
context on the structure of the accountability system, reducing the 
comparability of state systems and program results. The act pro- 
vided little guidance about the nature of the relationship that state 
agencies and local programs were to have in these new systems or 
about the state's role in providing technical assistance to local pro : 
grams. Some states have done little or nothing to elaborate this rela- 
tionship or build effective technical assistance procedures. In these 
cases, state discretion led to decisions that might not be considered 
to be in the spirit of the legislation or in the best interests of program 
improvement. 

The overall result of the legislative flexibility is mixed. In 1994, we 
found little evidence that states had created a dynamic program-im- 
provement culture based on outcome data, which many believe to be 
the intent of Perkins II. Furthermore, it appears to us that openness 
in the law, combined with strong state contextual factors, has length- 
ened substantially the timeframe i development and implementa- 
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tion. Even states that are moving in the right direction are moving 
more slowly than envisioned in the legislation. 

Unclear Integration and Coordination of Perkins II 
Provisions 

Although Perkins II appears to emphasize measures and standards, 
service to special populations, integrating academic and vocational 
curricula, and tech-prep education 1 equally, state agencies treated 
these as separate requirements and assigned them different priori- 
ties. The four states we visited differed in the emphasis they placed 
on developing and implementing performance measures and stan- 
dards relative to the other Perkins II initiatives. These differences in 
state priorities accounted for some of the differences we observed in 
progress toward systems of measures and standards. 

Similarly, Perkins II is unclear about how performance measures and 
standards are to be used to evaluate its other new priorities — service 
to special populations, tech-prep education, and the integration of 
academic and vocational education. Perkins II offers little guidance 
about how any of these activities are to be coordinated. Further- 
more, by giving states nearly two years to develop their performance 
measures and standards systems, Perkins II makes coordination 
more difficult. 2 While state administrators were developing systems 
of measures and standards, state and local administrators were be- 
ginning to implement tech-prep education programs and integration 
strategies. The two-year development phase for performance mea- 
sures and standards inhibited the use of these systems as evaluation 
tools for the other Perkins initiatives. 



'"The term 'tech-prep education program' means a combined secondary and post- 
secondary program vvhich leads to an associate degree or 2-year certificate; provides 
technical preparation in at least 1 field of engineering technology, applied science, 
mechanical, industrial, or practical art or trade, or agriculture, health, or business; 
builds student competence in mathematics, science, and communications (including 
through applied academics) through a sequential course of study; and leads to place- 
ment in employment." (The Carl D. Perkins Vocational and Applied Technology 
Education Act Amendments of 1990. P.L 101-392; Part E, Sec. 347.) 
2 

According to Rahn and Alt (1994), 45 states reported an increase in the amount of 
time spent developing performance measures and standards from 1990 to 1993 at the 
secondary education level. At the postsecondary level, 41 states reported an increase 
in 'ime spent in this area. 
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Ideally, each state's system of performance measures and standards 
would be used to assess the effectiveness of vocational programs, in- 
cluding the performance of special populations, tech-prep education 
students, and students in programs with integrated academic and 
vocational curricula. In fact, Erie included specific languag in its 
measures and standards to require their application to special popu- 
lations. One can imagine an ideal accountability system that 
provides information on the performance of tech-prep education 
students, youth apprenticeship students, and each special sub- 
population in a program. With such data, a local administrator 
would be able to compare the performance of subpopulations within 
a program, compare vocational programs within a school, and 
compare a particular program to overall school performance. More- 
over, this system would facilitate coordination between academic 
and vocational teachers, by providing measures of relevant academic 
and vocational skills on each student. Most importantly, compar- 
ative program information would allow administrators and instruc- 
tors to target program-improvement strategies. For example, if the 
tech-prep education program consistently exceeded all standards, an 
administrator might try to transform more vocational programs into 
tech-prep education programs. 

Unfortunately, the Perkins II priorities— measures and standards, 
service to special populations, integrating academic and vocational 
curricula, tech-prep education— are treated like disjointed programs, 
uncoordinated and at a different stage of implementation. In most 
states, Perkins II priorities are not part of a coordinated system at 
either the state or the local level, and performance measures and 
standards are not being used comprehensively to evaluate the other 
Perkins II initiatives. 

Lack of Models or Incentives for State and Local 
Implementation 

There is a major gap between development of measures and stan- 
dards (which usually occurred at the state level) and implementation 
of standards-based program-improvement systems (which must oc- 
cur at the local level). Development dominated state agency efforts 
in the four states we visited. Certainly, it is necessary to select mea- 
sures and standards and to create data systems to support them be- 
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fore these systems can be implemented. However, some state agen- 
cies had devoted so much attention to development that they had 
barely thought about subsequent stages of reform, such as how local 
administrators and instructors would use performance measures 
and standards to improve programs. 

The situation in one state reminded a researcher of the void that oc- 
curred following the birth of his first child. All of his and his wife's at- 
tention had been focused on the delivery (for instance, on childbirth 
classes and exercises) with little or no thought to what would come 
after. Once the child was born, it came as a sudden shock to Me par- 
ents that they now had to care for the child on a daily basis. It ap- 
peared that some states were in a similar position vis-a-vis Perkins II 
performance measures and standards. They focused extensively on 
the "birth" of measures and standards, but they devoted little time to 
thinking about implementing them or using them once they were 
created. States were unprepared for the next step, because the de- 
mands of the development stage overshadowed concerns about im- 
plementation. 

This shortsightedness may have occurred because Perkins II mod- 
eled a two-stage reform but did not provide two stages of incentives. 
The Office of Vocational and Adult Education monitored the sub- 
mission of state plans and the adoption of measures and standards, 
but there were no incentives for ensuring that performance mea- 
sures and standards would be used at the local level to improve pro- 
grams. Furthermore, while programs that did not make substantial 
progress toward the state standards for two years were required to 
develop a local improvement plan in conjunction with the state, 
there were no other explicit mechanisms in the law for monitoring 
local use of the measures and standards. 

For example, Erie adopted nine measures and standards (six of 
which were based on data that were already being collected!. Local 
agencies were required to select or develop measurement tools for 
the three new outcome areas. However, local administrators re- 
ceived no training in how to select appropriate instruments and have 
had very little contact from the state to see whether they were taking 
appropriate actions. Furthermore, the state has shown no interest in 
knowing whether they were using the measures and standards for 
program improvement. One postsecondary site responded by 
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initiating a campuswide effort to develop new measures, but another 
did almost nothing. With very little guidance from Jie state, local 
implementation depended almost entirely on individual initiative at 
the local level. 

Despite the lack of models or incentives provided in the legislation, 
one state agency succeeded in promoting the use of performance 
measures and standards at the local level, in part because the state 
administrator believed in the usefulness of performance data for lo- 
cal program improvement and seized the opportunity to implement 
such a system. The secondary vocational education agency in 
Piedmont adopted an approach in which local programs chose mea- 
sures from among several outcome areas and selected their own spe- 
cific measurement tools. Furthermore, the state agency provided 
training, developed guidebooks and report formats, and took other 
steps to make the local programs feel that their participation was im- 
portant. Although the state agency did not offer rewards or le 7 
penalties, it created an atmosphere that fostered local participation 
and, ultimately, implementation. 

Limited Expertise and Resources at tl e State Level 

The Perkins II performance measures ar d standards mandate cre- 
ated new responsibilities lor state staff, requiring them to develop 
expertise in new and often unfamiliar areas. At the same time, the 
act also reduced the set-abide for stat/j administration, to put more 
resources into direct services. This left some states in a bind: They 
lacked both the expertise to address new demands and the resources 
needed to develop it. 

Vocational education agencies in tht*se four states differed in the 
numbers of staff members; the sizes of their administrative budgets; 
the levels of expertise in th areas of psychometrics, statistics, and 
labor-market economics; and their capacities to implement local re- 
forms. In Erie, a single individual was responsible for statewide im- 
plementation at the secondary and postsecondary levels, in addition 
to other duties. This severely limited the time and energy that could 
be devoted to performance measures and standards. In other cases, 
staff did not possess the expertise needed to develop a program-im- 
provement system based on outcome data. For example, one state 
director commented, "My staff are in over their heads. There is not 
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one trained research person on staff. We wanted initially to conduct 
a labor market analysis, but we just didn't have anyone who knew 
how to do it." 

Some state agencies made performance measures and standards a 
high enough priority to allocate staff and funds to the effort; others 
devoted only limited resources to this area. Moreover, most states 
focused the bulk of their resources on developing their systems of 
measures and standards. Staff in some state education agencies 
complained about not having enough funds to provide the technical 
assistance necessary to implement performance measures and stan- 
dards effectively. Our site visits revealed a continuing need for addi- 
tional staff training in using outcome data for program improvement 
and a growing demand for technical assistance in this area. 

Mandated Measurement of Learning Outcomes 

Perkins II requires states to measure academic learning gain and 
suggests that states adopt measures of job or work skill and compe- 
tency attainment as well. The scarcity of valid assessment tools in 
these areas led states to adopt alternatives that were less than opti- 
mal. For example, some states delayed or phased in their perfor- 
mance measures and standards, because these states lacked mea- 
surement tools in some areas. All four states we visited lacked one or 
more measurement tools needed to complete their systems. Some 
states assumed the responsibility for developing new instruments, 
while other states selected from among measurement tools that were 
available. All were deficient in some manner. Still other states chose 
from among available instruments but recognized their shortcom- 
ings and planned to revisit the choices in the future. In addition, the 
act does not include definitions for key terms, such as "basic" and 
"more advanced" academic skills, which has led to widely varying in- 
terpretations and approaches. As one state director put it, the per- 
formance measures and standards mandate is "putting the cart be- 
fore the horse." He would prefer developing industry skill standards 
first, then appropriate assessments, and only last a system of mea- 
sures and standards. 

Alcorn and Columbia used standardized tests of academic achieve- 
ment at the secondary level to measure basic academic skills. These 
tests have oeen criticized, because they do not measure the aca- 
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demic skills for which vocational programs should be held account- 
able and do not measure these skills in a manner suitable for voca- 
tional students. To address some of these concerns, one state was 
developing an academic assessment instrument specifically de- 
signed for vocational students. 

In Erie and Piedmont, state agencies delegated to local agencies the 
responsibility for selecting and/or developing academic assess- 
ments. This decentralized approach defers difficult decisions to local 
programs, which are less likely to have the expertise required to 
tackle difficult measurement problems. This strategy raises concerns 
about the appropriateness of locally selected tests and the technical 
quality of locally developed measures. Decentralizadon of mea- 
surement choices also precludes the direct comparison of program 
performance. 

None of the states in our study possessed statewide academic tests at 
the postsecondary level. The only postsecondary academic tests in 
these states were those used to place students into appropriate 
English and math courses at the time of enrollment. Piedmont and 
Erie delegated responsibility for selecting or developing measures of 
postsecondary academic skills to local institutions. The remaining 
two states used remedial course completion as a proxy for basic aca- 
demic gain. One used general education and related academic 
course completion as a proxy for advanced academic gain. In both 
cases, it was assumed that students who received a passing grade in a 
course had gained the skills and knowledge covered by the course. 

Course completion has been criticized as a measure of academic 
gains, because it does not directly compare student achievement be- 
tween separate times. Furthermore, institutions usually define 
"successful course completion" as receiving a final grade of D or 
better. None of the instructors interviewed believed that a student 
who received a D could have mastered the body of knowledge con- 
tained in a course. Another criticism of measures of course comple- 
tion relates to remedial courses. Some instructors felt it was inap- 
propriate to hold vocational programs accountable for student gains 



3 In a nationwide survey, more than one-third of all states at the postsecondary level 
reported using course completion as a proxy for academic gain (Rahn, Hoachlander, 
and Levesque, 1993). 
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in preprogram, remedial courses, while others noted that it was 
wrong to credit vocational programs for academic gains that occur in 
these courses prior to enrollment in a vocational program. 

FACTORS SUBJECT TO STATE AND LOCAL INFLUENCE 

The second set of factors affecting successful implementation of the 
Perkins II system of measures and standards flows from the contexts 
of individual states and localities. As a result, these factors are pri- 
marily responsive to state or local action. Without state or local ac- 
tion, the following factors may continue to exercise a dampening ef- 
fect on the results of federal policy: 

• Choice between central and local implementation strategies 

• Existence of centralized data systems 

• Availability ot state-level assessment tools 

• Ongoing educational accountability and program review mecha- 
nisms 

• Historical relationships among agencies 

• Efforts to reduce burdens on local program staff 

• Influence of key administrators. 

State responses to the Perkins II accountability initiatives have been 
influenced by local factors; some of these advanced the development 
and/or implementation of state performance measures and stan- 
dards systems and others impeded it. 

Choice Between Central and Local Implementation Strategy 

A key factor that contributed to differences in state implementation 
is the division of responsibility between state and local agencies for 
developing and implementing measures and st^idards. Centralized 
approaches tended to be more efficient in the short term, but de- 
centralized ones may have greater long-term potential for local pro - 
gram improvement. 
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In this study, state vocational education agencies with a history of 
centralized decisionmaking selected outcome areas, stipulated spe- 
cific measurement tools, set standards, and identified criteria that 
would trigger local improvement plans. In contrast, states with a 
history of strong local control tended to confine their roles to select- 
ing general measurement areas and defining broad standards. Local 
educational agencies and postsecondary institutions were given re- 
sponsibility for selecting specific measurment instruments and for 
deciding whether their institution was making sufficient progress 
toward meeting state standards. In some cases, the approach to 
performance measures and standards differed at the secondary and 
postsecondary levels in the same state. 

To illustrate, in Alcorn, the state staff (with input from the committee 
of practitioners) selected measurement areas, developed instru- 
ments, and established overall state standards and aggregate per- 
formance goals for the whole system. For example, academic 
achievement at the secondary level was to be measured using a stan- 
dardized statewide test, and employment outcomes at both the sec- 
ondary and postsecondary levels were to be measured using state 
unemployment insurance data during particular quarters following 
program completion. In a similar vein, Columbia required locai re- 
cipients either to use state-developed measurement tools or to use 
state-defined methods of calculating performance on locally chosen 
measures. 

In contrast, Piedmont and Erie, with stronger traditions of local con- 
trol, delegated responsibility for developing or selecting some or all 
measurement instruments to local recipients. (Standards were set 
centrally in these two states.) In Piedmont, local school districts 
were allowed to develop their own academic and occupational com- 
petencies and measurement instruments or to adopt state-devel- 
oped ones, while districts were required to administer their own 
program follow-up surveys. In the four states we examined, where 
local agencies were responsible for development and/or implemen- 
tation of some measures and standards, more delays, confusion, and 
technical problems occurred than in states where these decisions 
were made centrally. 

Although systems with increased local responsibility faced early dif- 
ficulties, one state may be poised to achieve greater success because 
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of the investments the state made in local expertise. In contrast, the 
early successes of systems with more centralized control may dimin- 
ish if steps are not taken to engage local administrators in the pro- 
gram improvement process. For example, the data generated by the 
centralized performance measures and standards system in Alcorn 
were rarely being used by instructors to improve their course offer- 
ings. Although some district and institution administrators were 
using the data to identify broad problem areas, instructors stated 
that the data were too voluminous, complicated, and general to pro- 
vide them with the information they needed and desired to improve 
their programs. Rather than numerous pages of tabular data, in- 
structors preferred a simplified presentation format, and rather than 
an overall employment rate for program completers, they desired 
information on the specific employers who had hired their com- 
pleters, on the specific jobs those completers had taken, and on the 
skills they were using or needed in those jobs. Furthermore, several 
administrators had not yet passed the performance measures and 
standards data on to their instructors, believing the instructors 
would be overwhelmed, and some spoke candidly that they did not 
expect their instructors ever to use the data. 

In contrast, at the secondary level in Piedmont, the highly decentral- 
ized performance measures and standards system led to widespread 
use by instructors of related academic assessments and the perfor- 
mance data generated by them. Because these assessments had 
been developed locally with support and encouragement from the 
state, instructors found them relevant for assessing student out- 
comes and improving programs. Furthermore, instructors and ad- 
ministrators found the performance report formats supplied by the 
state useful and understandable. 

These examples illustrate strong differences in local use of perfor 
mance measures and standards related to the degree of centralized 
or decentralized development, but the differences are not always so 
great. A centralized approach may be more efficient, may lead to 
more elaborate and well-organized data, and may provide greater in- 
terprogram comparability. A decentralized approach may help en- 
sure that performance measures and standards are relevant locally 
and may lead to greater local participation in and commitment to the 
program review and improvement process. 
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Existence of Centralized Data Systems 

Statewide student databases and data processing capabilities af- 
fected state responses to Perkins II performance measures and stan- 
dards initiatives. Agencies with centralized data systems were more 
likely to assume a strong role in collecting and reporting perfor- 
mance measures and standards than those without such systems. 
Although this is not always true, states with more centralized record- 
keeping experience and the computer capacity to maintain and ana- 
lyze large-scale databases were more likely to construct centralized 
data systems in response to Perkins II. 

For example, in Alcorn and Columbia, the state vocational education 
agencies obtained data for some performance measures from other 
educational departments or state agencies, compiled those data, and 
reported them to local school districts and community colleges. For 
other measurement areas, local administrators reported data to the 
state vocational education agency, which then summarized perfor- 
mance on the measures and standards and returned the data to their 
originators in a revised form. In both cases, the state vocational edu- 
cation agencies played an important role in either collecting or re- 
porting the data. As a consequence, even though many of the data 
were locally generated, administrators and instructors in these states 
viewed performance measures and standards data as belonging to 
the state. One postsecondary dean said, "The data don't match ours, 
but this lets us know how the state views us." 

In contrast, at the secondary level in Piedmont, the state vocational 
education agency played only a minimal role in data collection and 
reporting. Local school districts compiled their own data, tallied the 
appropriate statistics, and reported them to the state agency on 
standard report forms that had been supplied by the state. Local 
administrators and instructors we interviewed in this state were 
more knowledgeable about and interested in the performance mea- 
sures and standards than local staff in other states. 

A second consequence of a centralized data system is that it con- 
tributed to states focusing on the data more than how the data could 
be used. For example, Alcorn's data-computing capacity led state- 
level vocational education staff and the committee of practitioners to 
envision a sophisticated system of performance measures and stan- 
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dards that would provide local school districts and community col- 
leges with a broad range of outcome data at many levels of aggrega- 
tion, including district, school or institution, program, and special 
populations' status. However, while devoting substantial time to de- 
veloping this data system, stat ^ staff spent less time thinking through 
how the data would be used at the local level to improve programs. 
As one state-level administrator stated, "We've provided them with 
all the data; now all they need to do is use it" Although the system 
succeeded in generating hundreds of pages of data per school or 
community college district, there was little evidence that the data 
were being used at the local level, particularly by instructors. At the 
end of the second year of implementation of performance measures 
and standards, state-level staff still placed a higher priority on im- 
proving data quality than on improving local use for program eval- 
uation and improvement. 

In contrast, secondary-level state staff in Piedmont had minimal ca- 
pacity to collect and manipulate data. They devoted their time to de- 
veloping guidelines for local agencies to follow and to providing 
other technical assistance to support local development efforts. 

Availability of State-Level Assessment Tools 

As noted above, certain provisions of Perkins II create assessment 
problems for all states — for example, the requirement that state sys- 
tems include measures of academic learning gains has been a uni- 
versal tumbling block— and it is appropriate for the federal govern- 
ment io take steps to address these concerns. However, other 
aspects of assessment affect the implementation of measures and 
standards at the state and local levels in ways that may not be ad- 
dressed best by federal action. In particular, the interplay between a 
state's choice of measures and standards and the state's existing 
testing system affects the implementation of the measures and stan- 
dards. R r thermore, states that delegated decisions about vocational 
measures to local agencies added another layer of sensitivity to state 
and local assessment experience and practice. 

States' actions in implementing measures and standards were af- 
fected by the prior existence of relevant state assessment tools. 
Where such measurement tools existed, states generally incorpo- 
rated them into their performance measures and standards systems, 
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although the use of existing measures had mixed results. For exam- 
ple, Columbia and Alcorn relied on existing statewide standardized 
tests of academic achievement to satisfy the requirement to measure 
"learning and competency gains, including student progress in the 
achievement of basic and more advanced academic skills" at the 
secondary level. These existing academic tests served as readily 
available measures, but they also had several limitations. For one, 
tests in both states assessed academic attainment in early high 
school— neither test had the capacity to measure the gains students 
achieved throughout their high school years. Both state agencies 
planned to phase in an additional test to measure gains, but neither 
had yet done so. Second, these standardized academic tests were not 
designed to measure academic skills specifically related to occupa- 
tional competencies, which many vocational educators have argued 
should be the focus of Perkins II performance measures and stan- 
dards. Third, instructors do not always find results relevant to their 
courses. Currently, neither state plans to develop tests of occupa- 
tionally related academic skills. 

In contrast, Piedmont's secondary-level state staff focused on help- 
ing instructors develop measures of the relevant academic skills and 
occupational competencies appropriate for specific local programs. 
The state also sponsored the development of statewide prototype 
tests that instructors could choose to implement in lieu of develop- 
ing their own local tests. Although developing these tests took sub- 
stantial time and staff resources and although questions of test va- 
lidity and reliability remain to be addressed, Piedmont's vocational 
educators were able to produce tests that measured both student 
gains and related academic skills. 

No statewide measure of academic performance existed in Erie, so 
state staff required local recipients at both the secondary and post- 
secondary levels to choose or develop appropriate measures of aca- 
demic and occupational skill gains. Although state staff identified 
the skills that should be measured in general terms, they provided 
little assistance to local agencies in selecting appropriate assess- 
ments. Some staff questioned whether the instruments that had 
been selected by some local agencies would adequately assess the in- 
tended skills. 
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At the postsecondary level, none of the states in our study possessed 
a statewide measure of academic skills. Although Erie required local 
recipients to seek or develop an appropriate measure, the other three 
states measured academic gains through course completion. For ex- 
ample, states counted successful completion of developmental or 
remedial courses or of general education or related academic courses 
as academic gains. However, state-level staff in Alcorn were develop- 
ing tests of basic reading and mathematics skills at the postsec- 
ondary level. 

The lack of relevant instruments for measuring academic and occu- 
pational competency led states to adopt alternative strategies for 
measuring learning outcomes. These alternatives often were not the 
most appropriate. Furthermore, difficulties in measuring academic 
and occupational skills forced states to slow implementation of this 
portion of the Perkins II mandate. In a few cases, Perkins II require- 
ments spurred states to develop their own instruments, which ap- 
peared to be suitable at first glance. 

Ongoing Educational Accountability and Program Review 
Mechanisms 

States' reactions to Perkins II were strongly influenced by the educa- 
tional accountability context that existed within the states. Some 
states had strong accountability mechanisms that attached incen- 
tives or sanctions to educational performance, although weaker ac- 
countability systems were more common. All states had program 
review and accreditation procedures designed to monitor and/or 
certify educational quality. Perkins II measures and standards served 
a similar purpose, and some states tried to consolidate Perkins II ac- 
countability with existing efforts. However, this was not always the 
case, and it is interesting to explore the degree to which states tried 
to integrate these activities. 

Erie had established an accountability system based on student out- 
comes for both secondary and postsecondary vocational education 
several years prior to passage of the act in 1990. In response to 
Perkins II performance measures and standards requirements, the 
state added three new measures to the six they had already been 
collecting. By folding performance measures and standards into an 
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existing accountability system, the state was able to respond quickly 
to die Perkins II mandates. It is also possible that by attaching 
Perkins-specific measures to an established system, local vocational 
administrators and instructors would be more likely to incorporate 
them into their local accountability processes. However, there was 
little evidence that the existing accountability system was being 
tninsformed into the program improvement process envisioned by 
Pe/kins. 4 

Alcorn already had several components of an accountability system 
for elementary and secondary education in place prior to Perkins II. 
Specifically, the state had been issuing school report cards for a 
number of years and had recently initiated a review process whereby 
schools regularly evaluated their performance in state-specified ar- 
eas based on locally selected measures. However, neither the report 
cards nor the review process incorporated outcome measures rele- 
vant to vocational education. While state vocational education staff 
members have begun a dialogue with general education staff to bring 
Perkins II efforts closer to other strictly academic accountability ef- 
forts, by 1994, performance measures and standards had been devel- 
oped and implemented largely in isolation from the general ac- 
countability initiatives. 

Previous experience with statewide school report cards had both 
positive and negative effects on the development of Perkins II mea- 
sures and standards in Alcorn. On one hand, state and local ad- 
ministrators, instructors, and members of the committee of prac- 
titioners were already familiar with the idea of using outcome data to 
evaluate institutional performance, which may have facilitated 
development of a system of performance measures and standards. 
On the other hand, this same experience may have led state staff to 
ignore how the data would realistically be used locally to improve 
programs. As an accountability tool, the school report card was used 
largely by school administrators as "a public relations piece," in the 
words of one superintendent, primarily in discussions with school 



Early evidence indicated that, without follow-through and assistance from state staff, 
local programs were largely ignoring the requirement for new measures. Because the 
state had been responsible for collecting the six extant measures and had done little to 
encourage use of the data, local administrators were paying scant attention to the 
measures as well. 
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boards and parents. Instructors paid little attention to these data, 
because they were not provided in a form useful to them and 
because they were used to make comparisons between districts the 
instructors felt were unfair. As with the school report cards, state- 
level staff passed the performance measures and standards reports 
down through the secondary and postsecondary education hier- 
archies, expecting them to reach instructors who would then use the 
information to improve programs. However, by the end of the 
second year of implementation of performance measures and stan- 
dards, few instructors had ever seen the data, and there was wide- 
spread skepticism among local administrators that instructors would 
ever use them. 

In addition to these accountability initiatives, states also had other 
evaluation and program review systems in place prior to Perkins II. 
Most state vocational education agencies followed a multiyear cycli- 
cal program review process at both the secondary and postsecondary 
levels. State-level vocational education staff indicated that perfor- 
mance measures and standards data would likely be incorporated 
into the multiyear program review process. 

Additionally, all postsecondary institutions were subject to periodic 
reviews by independent regional accreditation agencies. While the 
cyclical program reviews traditionally focused on process measures, 
such as vocational enrollments and sex-equity issues, the regional 
postsecondary accreditation agencies have recently begun moving 
toward outcome-based evaluations. Evidence in several states indi- 
cated that Perkins II performance measures and standards were 
taking a back seat to the accreditation process at the postsecondary 
level, particularly in those institutions that recently went through the 
new outcome-based accreditation process. In part because accredi- 
tation is crucial to the viability of an institution, accreditation re- 
quirements were receiving greater attention than Perkins II. 
Furthermore, the accreditation process is generally well regarded by 
postsecondary institutions, because it permits them to evaluate 
themselves in terms of their own goals. In contrast, some feel that 
Perkins II performance measures and standards are centrally dic- 
tated without appropriate institutional contribution. However, since 
accreditation reviews sometimes occur as infrequently as once every 
ten years, the annual Perkins requirements may gain in importance 
over time. 
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Other federal accountability requirements also influenced states' re- 
sponses to Perkins II. For example, recent federal student right-to- 
know legislation requires that postsecondary institutions receiving 
federal financial aid track the educational experiences of a cohort of 
full-time students. At least one state in our sample tried to consoli- 
date requirements by using this same cohort in its Perkins II perfor- 
mance measures and standards system. However, because first- 
time, full-time students are often a minority at community colleges, 
instructors generally found the performance data they received use- 
less for program improvement For example, one Piedmont instruc- 
tor had only three students in the selected cohort (and one of these 
had been sent to prison). The instructor did not believe that the re- 
sulting 33 percent incarcerated outcome rate accurately reflected his 
program performance and did not find that it provided him with 
useful information for improving his program. 

In summary, the relationship between existing accountability initia- 
tives and development of Perkins II performance measures and 
standards was complex. Combining these various efforts may ulti- 
mately bolster them all. However, Perkins II emphasizes program 
improvement, while many of the other initiatives do not. This em- 
phasis may be lost if Perkins II accountability is joined with systems 
that use performance data for other purposes. Furthermore, where 
federal regulatory demands are in conflict, practical constraints may 
force states to satisfy one requirement at the expense of another. 

Historical Relationships Among Agencies 

Existing governance structures and collaboration patterns among 
state agencies affected states' responses to Perkins II requirements. 
For example, unified secondary and postsecondary vocational edu- 
cation agencies or agencies that were housed in the same state office 
produced coordinated Perkins II efforts: Erie agencies adopted a 
single set of performance measures and standards for both levels of 
education, while Alcorn agencies adopted parallel sets of measures 
and standards and worked together to develop new outcome data. 
In contrast, Columbia secondary and postsecondary vocational edu- 
cation agencies historically acted separately, and they developed 
wholly different and separate performance measures and standards 
systems. In Piedmont, the secondary vocational education agency 
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took the lead in developing its performance measures and standards 
system, after which the postsecondary agency scaled down the sec- 
ondary system for its own implementation. 

In addition to historical relationships between secondary and post- 
secondary vocational education agencies, responses to Perkins II re- 
quirements were also affected by existing relationships between 
these agencies and those responsible for general education, the Job 
Training Partnership Act (JTPA), and other workforce programs. For 
example, the Alcorn state vocational education agency operated 
largely separately from the elementary and secondary education 
agency and the community college board. They had to establish 
formal links to obtain data from these agencies, a process that was 
facilitated because of historically good relations among them. In 
some cases, the federal performance measures and standards initia- 
tive provoked new inter-agency collaboration. For example, the 
Perkins emphasis on placement outcomes spurred the Alcorn voca- 
tional education agency to collaborate with the state's JTPA agency, 
which had substantial experience using state unemployment insur- 
ance data for obtaining employment and earnings outcomes. 

Efforts to Reduce Burden on Local Administrators 

Reducing the data-collection and reporting burden on local adminis- 
trators was foremost in the minds of many state vocational education 
agencies when they designed their systems of measures and stan- 
dards. This approach appeared to expedite the selection of perfor- 
mance measures and standards and the collection and reporting of 
data and to increase initial acceptance of performance measures and 
standards by local administrators. However, efforts to reduce the 
burden of measures and standards may also reduce the effectiveness 
of the system in the long run by limiting local familiarity and "buy- 
in." Moreover, the focus on burden reflects a narrow perception of 
the system of measures and standards as a reporting exercise, not as 
an essential core activity in a meaningful program management and 
improvement system. 

Three of the four states we visited attempted to design their mea- 
sures and standards to minimize data-collection demands on local 
institutions. In Columbia, state staff at the postsecondary level 
"made every effort" to adopt measures for which data were already 
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being collected by local institutions. In Erie, six of nine measures 
adopt ci v/ere already being collected. Similarly, Alcorn, adopted 
measure* for which local schools already collected data or measures 
that could be satisfied by data obtained through another state 
agency. All were proud of these actions, and felt they expedited the 
process of developing their state systems of measures and standards. 

On the other hand, reducing the burden of data collection and re- 
porting also reduced local awareness of the reforms, particularly for 
instructors. Well into the second year of implementation of Perkins 
II measures and standards, few instructors in Columbia, Erie, and 
Alcorn had either heard of performance measures and standards or 
understood what their role was to be in using them. Furthermore, in 
at least one case in which performance measures and standards data 
had been fed back to administrators or instructors, few found the 
data relevant or felt committed to using them. 

In sharp contrast, high school instructors in the sites we visited in 
Piedmont responded well to the "burden" that was placed on them 
to develop and implement local assessment of related academic 
skills and occupational competencies. While some local personnel 
initially resisted the new requirements, most became more positive 
with time and with continued encouragement and support from the 
state. In this case, local educators who were invited to participate in 
the process and now have greater ownership over the results re- 
sponded well to the additional responsibilities placed on them by 
federal or state mandates. Furthermore, local educators who were 
involved in data definition, instrument development, and the like 
were in a better position to use the data when they were produced. 
Protecting local educators from the burden of developing a new sys- 
tem may ultimately lessen their capabilities and incentives to use 
them. 

Influence of Key Administrators 

The presence or absence of strong leadership affected the nature of 
states' responses to Perkins II and the progress states made in devel- 
oping and implementing performance measures and standards. In 
most of the states, a single individual had primary responsibility for 
the implementation of performance measures and standards at the 
secondary or postsecondary level (or both). The commitment of and 
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approach used by this key administrator largely determined the 
progress made by the state. At one extreme, some key state voca- 
tional education staff acted decisively to develop expanded systems. 
For example, a key administrator in Columbia took advantage of 
Perkins II mandates to promote reforms she had long wanted to im- 
plement. At the other extreme, some key administrators took only 
those few steps that were necessary for compliance with the law. 

These four states revealed dramatic contrasts in the speed and effec- 
tiveness with which administrators implemented their statewide sys- 
tems. Sometimes, secondary and postsecondary systems within the 
same state varied dramatically. For example, the primary state-level 
administrator for secondary vocational education in one state was an 
active and effective change agent who seized upon the Perkins II 
mandate as an opportunity to promote greater local accountability, a 
cause in which he believed strongly. He supplied a vision of how the 
reform would work, rallied enthusiasm for it, and provided support 
to those who were responsible for enacting the changes. In contrast, 
the administrator for postsecondary vocational education in the 
same state did very little to encourage local implementation, stating 
candidly that he expected the performance measures and standards 
requirements to go the way of many national initiatives and be for- 
gotten within a few years. 

In another state, the vocational coordinator took the performance 
measures and standards mandate seriously, but kept his role to a 
minimum. This key administrator communicated with local staff 
about Perkins II requirements primarily in writing, assuming that 
school-level staff would take his general guidelines and use their own 
initiative to select and implement several new performance mea- 
sures. However, since the state devoted minimum attention to 
training and support, few schools took steps to implement the new 
measures. Some local administrators called colleagues to find out 
what they were doing in response to the state guidelines. When they 
learned that others were waiting for further direction before acting, 
they held up their own actions, effectively putting the entire initiative 
on hold. 

Although leadership styles varied widely, most of the administrators 
we interviewed actively promoted the implementation of perfor- 
mance measures and standards. However, the power of individuals 
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to influence the development and implementation of these systems 
in both positive and negative ways was demonstrated repeatedly. 

FUTURE CONCERNS 

Our ability to predict is limited, but some potential problems already 
seem likely. The following issues and events are likely to have a ma- 
jor impact on statewide systems of measures and standards in the 
near future. They deserve careful consideration in federal and state 
vocational education planning. 

Skills Standards and CurricUum Standards 

The Departments of Education and Labor are funding the develop- 
ment of industry skill standards in more than 20 occupational areas 
and curriculum standards in six educational fields. What will these 
efforts mean for statewide systems of measures and standards? 
Certainly, valid national standards deserve consideration by states. 
They might provide invaluable common criteria for occupational 
preparation. However, no one knows how best to accommodate 
such standards in systems of measures and standards, or how diffi- 
cult (or expensive) such coordination might be. What supplemental 
work will be required to integrate industry standards and curriculum 
standards with state systems (e.g., connection between state occu- 
pational task lists and industry standards)? How will this work be 
supported? 

Not only will the development of industry skill standards have an ef- 
fect on statewide systems of measures and standards, but the organi- 
zations developing the industry standards need to be informed about 
the Perkins II measures and standards systems used in vocational 
education. The key question is, what effect should states' systems of 
measures and standards have on the structure of the industry skill 
standards? It is important that those developing the industry stan- 
dards be aware of the accountability mechanisms states have been 
creating under Perkins II. 
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Data Quality 

From 1991 to 1994, states devoted most of their energies to selecting 
and adopting measures and standards and developing data systems 
to support them. As yet, little attention has been paid to the quality 
of the measures and standards. 0 As soon as statewide systems be- 
come operational and programs are required to judge their perfor- 
mance against state standards, questions of reliability and validity 
will become more important. In states that adopt high stakes for per- 
formance (as in the case of Columbia), potential data quality prob- 
lems will be exacerbated, because high stakes are known to corrupt 
the interpretation of educational data. The need for evaluations, of 
data quality will increase once systems are fully implemented. 
However, states are unlikely to have the resources or expertise neces- 
sary to conduct these investigations. If these issues are rjot resolved, 
the whole enterprise may suffer, so it is important for the federal 
government to develop mechanisms to facilitate assessment of the 
quality of measures and standards. 

Integration of Academic and Vocational Education 

Potentially the most far-reaching reform embodied in Perkins II is 
the integration of academic and vocational education. This move- 
ment raises several problems for defining, measuring, and attribut- 
ing outcomes and therefore threatens the validity of existing systems 
of measures and standards. First, states are unlikely to have included 
integrated outcomes in their current systems, so major changes will 
be required. Second, the development of measurement tools for in- 
tegrated outcomes will probably lag behind the development of cur- 
riculum. Third, the conception of a vocational program will have to 
be changed to include academic components. Fourth, it will be diffi- 
cult to attribute outcomes to specific courses and to use outcome 
data for program improvement. Although fully integrated programs 
are many years away, many states already have begun implementing 
some integrated academic curricula, and these states are already 
facing problems defining and measuring outcomes. Early attention 



This is one of the questions being addressed by the Batelle study of Perkins II. 
However, that study relies on data collected in 1993, before most state systems were 
operating on a regular basis. 
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to the questions raised by integrated academic and vocational edu- 
cation is warranted. 

Consistency in Federal Vocational Education Policy 

Some state vocational educators explained their less-than-enthusi- 
astic response to Perkins II by saying, "It's going to go away." They 
believe new laws will supplant many of the initiatives contained in 
Perkins II. When implemented believe mandates are only tempo- 
rary, they reduce their efforts and adopt a mind-set of minimum 
compliance rather than commitment. Unfortunately, the volatility of 
federal vocational education policy discourages rapid and effective 
response to federal initiatives. Long-term change is unlikely under 
these circumstances. 

Perkins II measures and standards embody a radically different ap- 
proach to accountability. It may take five years of implementation 
for a fair test of this initiative. However, there already is talk within 
the states that drastic changes in the regulations are imminent. As a 
result, some administrators are curbing their efforts; many aiX; pro- 
ceeding with caution. To promote effective implementation of mea- 
sures and standards, the federal government must make it clear that 
it intends to stay the course. Further, it must give vocational educa- 
tors ample time to develop, implement, and fine-tune the system be- 
fore making judgments about its efficacy. A minimum of five years is 
required to see whether outcome-based local program improvement 
in the Perkins II model will work or not. 
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RECOMMENDATIONS 



This research suggests specific actions federal policymakers should 
take to improve the implementation of Perkins II measures and 
standards. Although our case studies were limited to four states, 
both our informal conversations with other vocational educators and 
previous research on the implementation of Perkins II meesures and 
standards suggest that these concerns are widespread enough to 
warrant such actions. Potential responses include changes in regu- 
lations, investments in the solution of common problems, and new 
capacity-building initiatives at the state and local levels. These rec- 
ommendations follow directly from the factors identified as being 
subject to federal influence in the previous chapter. 

CLARIFY THE INTERRELATIONSHIPS AMONG PERKINS II 
MANDATES AND THE COORDINATION OF PERKINS II 
INITIATIVES 

Perkins II requires that states emphasize systems of measures and 
standards, the integration of academic and vocational education, 
tech-prep education, and service to special populations, but it does 
not tell states how these activities should be interrelated, nor does it 
suggest how states should coordinate their efforts. As a result, states 
implement activities in isolation and assign them different priorities. 
For example, some put tech-prep education on hold while focusing 
on measures and standards; others assign their most talented staff to 
tech-prep education while delegating measures and standards to one 
individual. 
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Perkins II gives little direction to states about how to coordinate their 
efforts on measures and standards, tech-prep education, integration 
of academic and vocational education, and special populations. The 
omission of models for coordination decreases the potential effec- 
tiveness of the act, particularly the effectiveness of the systems of 
measures and standards. States need guidance regarding how to 
make their systems of measures and standards support their efforts 
to integrate academic and vocational education, create tech-prep 
education programs, and promote service to special populations. 
For example, performance measures and standards could be seen as 
a mechanism for evaluating all the Perkins II priorities. At present, 
states do not always see how the required elements can be interre- 
lated, e.g., how needs assessment fits into the state plan, how the 
state plan fits with the local application process, and how the local 
application process fits into the annual review cycle. Neither do they 
see how these are all elements of an evaluation system that applies to 
all the Perkins II initiatives. Instead, there is wide variation in the 
relative priority assigned to these four critical initiatives and little co- 
ordination. Changes in the language of the legislation could clarify 
these relationships. 

CREATE MODELS FOR OUTCOME-BASED PROGRAM 
IMPROVEMENT 

To date, states have emphasized the "nuts and bolts" of creating 
systems of measures and standards. Over time, they need to build on 
this foundation and focus on using these systems for program im- 
provement. At present, most state action is still driven by the man- 
date to develop a structure for accountability, i.e., a system of mea- 
sures and standards. The law says little about how that structure is 
supposed to be used to make programs better. Sites must develop 
program improvement plans, but nothing suggests how one trans- 
lates outcome deficiencies into action plans either at the local level 
or at the state level if the joint planning provision comes into effect. 
We talked to one state administrator who had begun to tackle this 
question by developing a set of specific questions linked to the state's 
measures and standards for use at a site to organize its search for 
improvement. However, such thought about the use of measures 
and standards was rare. 
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One way to encourage states to build toward program improvement 
is to require them to develop models for using outcome-based in- 
formation for program reform. Certainly, it is time for states to think 
seriously about this phase of accountability. Yet, some states might 
find any additional requirements distasteful and many would argue 
that additional paperwork is counterproductive. The alternative we 
prefer would be to commission another agency to collect examples of 
effective practices and disseminate them widely for states and local 
programs to use. 

PROVIDE FOCUSED TECHNICAL ASSISTANCE REGARDING 
CHOICES AND RESOURCES 

Perkins II gives states far greater programmatic discretion while lim- 
iting the resources they can devote to administration. This places 
greater demands on state administrators, but limits their capacity to 
act. Under these circumstances, federal actions that helped states 
respond to their choices and make better use of resources might 
significantly improve implementation of the act. 

The "flexible mandates" contained in Perkins II placed demands on 
states and local agencies but offered no guidance in how to meet 
those demands. States were required to create systems of measures 
and standards with certain properties but were allowed considerable 
discretion in the design of these systems, including the choice of 
outcomes, the selection of measurement tools, and the setting of 
standards. Many lacked the expertise to make these decisions wisely. 
Furthermore, some states delegated many of these choices to local 
agencies 1 This cascade of delegated decisionmaking placed new 
and unfamiliar demands on state and local staff. Few had any 
previous experience implementing outcome-based accountability 
systems in vocational education. 

As a result, states looked to the U.S. Department of Education for as- 
sistance in clarifying the meaning of the regulations and providing 
expertise in solving problems of measurement, assessment devel- 
opment, and standard setting. Unfortunately, little help was offered. 



This led to considerable variation of measures and standards between states as well 
as large within-state variation in selected states. 
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The federal agency focused its resources on compliance monitoring 
(for all the provisions of Perkins II) rather than on providing guid- 
ance about measurement problems or acting energetically to in- 
crease a state's capacity to respond through training. Other agencies 
stepped in to try to address the most pressing problems (e.g., NCRVE 
workshops on selecting measures and setting standards). This sup- 
port was helpful, but it did not satisfy the demand. Furthermore, the 
demand for technical assistance still exists, as many states lag behind 
the four we visited in implementing their systems, and some of the 
more difficult measurement issues remain unresolved. 

A further demand for technical assistance arose within those states 
that delegated the choice of measures to the local level. Those with 
ultimate responsibility for deciding on measurement tools and pro- 
cedures lacked the expertise to make these decisions and looked to 
the state as well as to their colleagues in other districts for assistance. 
The states' record in responding is mixed. Piedmont provided ade- 
quate technical assistance by training district vocational education 
administrators, who then trained program instructors. Erie did not. 

Furthermore, we predict that a new set of technical assistance needs 
is on the horizon in the area of program improvement. Statewide 
systems will be operational on a large scale in the 1994-95 school 
year, and local programs will begin to compare their performance to 
state standards. Few at the local level have any experience using 
outcome data for creative program planning, so programs that do 
not make the grade will look to their states for guidance with pro- 
gram improvement. States, in turn, will turn to the federal govern- 
ment for assistance, for they too lack expertise in using performance 
measures as program-improvement tools. 

The federal government should act now to address these needs by 
creating a mechanism for providing relevant technical assistance to 
states. In the long run, the goal should be to build capacity at the 
state level for using the standards framework as a program improve- 
ment tool. One approach might be to press for greater integration of 
vocational education measures and standards with existing state 
program review procedures. Because many states have the same 
questions, we believe the most efficient way to provide these services 
is centralized, but we would not preclude other options. Whatever 
the form, resources should be devoted to solving measures-and- 
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standards problems and increasing the capacity of states to make 
their systems work. One state coordinator described his experience 
with statewide measures and standards as "trying to build a 747 in 
flight." Trying to do this without appropriate technical assistance 
and support is a prescription for failure, and much more could be 
done to promote success. 

Resource constraints exacerbate the problem. Perkins II places 
greater demands on state agencies than its predecessor, but restricts 
the use of funds for state-level services. This comes at a time when 
many state administrative budgets are also being reduced. State 
administrators believe they lack the resources necessary to respond 
to Perkins II mandates effectively. These limitations are particularly 
severe at the postsecondary level, where some states allocate very 
small proportions of their Perkins II funds. Some efficiencies are 
possible through better coordination of efforts on measures and 
standards, tech-prep, integration, and special populations, and fed- 
eral efforts to solve difficult measurement problems will also lessen 
the demands on state agencies. Furthermore, states could be en- 
couraged to increase local participation in the technical assistance 
process over time, substituting regional networks and collegia! assis- 
tance for some of the centralized expertise required initially. 

Other resource problems might be addressed by increasing adminis- 
trative allocations for certain activities. We are not advocating a re- 
turn to the previous model, which many felt diverted too many dol- 
lars from students to central bureaucracies. As we see it, the need for 
supplemental resources and for technical assistance is not constant 
and can be fulfilled in many ways. Therefore, the response can be 
flexible. State departments of education need additional resources 
during times of initial development We suggest increasing the funds 
available for state administration during the start-up phase, so states 
can meet initial demands and develop some of the expertise they will 
need to operate reformed vocational education systems. 

ADDRESS COMMON MEASUREMENT PROBLEMS 

One reason the implementation of Perkins II has been so challenging 
to states is that the vision embodied in the guidelines exceeds the ca- 
pacity of existing measurement tools. In particular, the technology 
to measure learning and occupational performance in reliable, valid, 
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and efficient ways is not widely available. For example, although 
much work is being done to develop performance measures of occu- 
pational skills, this is an emerging assessment area with few opera- 
tional systems. The situation is worse in some areas, e.g., measuring 
gains in advanced academic skills appropriate for the postsecondary 
level, where none of the four states found existing measurement 
tools or had the expertise to develop their own direct measures. 

Most states are not equipped to develop tools for measuring learning 
and occupational outcomes, and it is unfair to drop this difficult task 
in the lap of state vocational educators. The federal government 
needs to take responsibility for addressing these problems, since they 
are largely the result of the provisions of Perkins IL 

At the same time, the federal government is supporting a variety of 
new assessment initiatives whose relationships to Perkins II mea- 
sures and standards are unclear. These range from the development 
of moi -2 than 20 sets of industry skill standards to the establishment, 
of natis>nal goals and curriculum standards for academic education. 
There should be some connection between the assessment demands 
of Perkins II and the assessment development efforts being funded 
separately. At a minimum, states need guidance about how the skill 
standards should relate to their statewide systems. Other assessment 
development projects to meet the specific needs of Perkins II should 
also be considered. 

CONCLUSIONS 

The 1990 amendments to the Carl D. Perkins Vocational and Applied 
Technology Education Act entailed major changes in federal voca- 
tional education priorities and practices. Among the most visible in- 
novations in Perkins II is the creation of statewide systems of mea- 
sures and standards of performance. The purpose of these systems is 
to create an objective, outcome-based framework for program evalu- 
ation that can be used as the cornerstone of local program improve- 
ment efforts. We found that states had made much progress creating 
the structure of measures and standards but had devoted much less 
attention to creating a process for using standards as program-im- 
provement tools. We also identified a number of factors that affected 
state actions on measures and standards, many of which reduced the 
likelihood that state systems would function as envisioned in the law. 
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Some of these factors are subject to federal intervention, and we 
identified steps the federal government could take to ameliorate the 
situation. Incorporating these changes into the reauthorization of 
federal vocational education will increase the efficacy of statewide 
systems of measures and standards. Legislators also should antici- 
pate difficulties and conflicts that may be created by pending reforms 
of education, training, and workforce preparedness programs, and 
they should work to coordinate the accountability requirements of 
all these initiatives. 
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RAND/MPR INTERVIEW GUIDE 
Secondary/Post-Secondary Differences 

Clarify the appropriate "level(s)" for the interview. Is the respondent 
knowledgeable about secondary vocational education, post- 
secondary, or both? Is s/he willing to discuss secondary issues, post- 
secondary or both? [Adjust questions accordingly.] 

Nature of statewide measures and standards 

I mailed MPR's summary description of [state's] standards and 
measures and asked you to update it to reflect any errors or changes. 
[Show document] Are any revisions necessary? 

Are these measures or standards likely to change this year? If 
so, how? 

Are there plans to review or revise the measures I the standards 
in the future? If so, what will be done? 

States can use their measures and standards for many purposes, for 
example, evaluating instructor effectiveness; which purposes are 
most important in [state] at the secondary / post-secondary level? 

(Other purposes: monitoring statewide performance, holding 
programs accountable for outcomes, allocating resources, 
providing individual student information) 

Can you rank the purposes in order of importance? How do 
you know? 

Importance of measures and standards vis-^-vis other 
vocational education initiatives 

How important are measures and standards relative to other 
vocational education efforts? 

— relative to other Perkins requirements, such as Tech Prep, 
integration of academic and vocational education, special 
populations, etc.? 
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— relative to other state-initiated vocational efforts? 

— important in terms of actions (e.g., staff activities, resource 
commitments) 

What kinds of activities have occurred? Can you give examples? 

— important in terms of lon[ -range impact (e.g., broader reform 
agenda) 

Which cfianges will be the most lasting? Why do you think this is so? 

Note: Clarify secondary / post-secondary differences 

Clarify importance in terms of the interests of special 
populations 

Would [state] have taken these actions if there had not been a federal 
requirement? 

What evidence is there that [state] was moving in this 
direction on its own? 

Are measures and standards supportive of or antagonistic to other 
vocational education reform efforts? 

Which other vocational initiatives? How are they 
complementary or how do they conflict? 

Integration of vocational education accountability with other 
educational reforms 

Looking at educational reform more broadly, what are [state's] 
current reform efforts in general and academic education? 

How do these initiatives relate to accountability in vocational 
education? 

For example, 

Is accountability an important issue in general and academic 
education? What is being done to promote accountability? 
How have vocational education measures and standards 
affected this process? Why or why not? 
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Is reform of assessment an important item in general and 
academic education? What assessment reforms are occurring? 
How do they relate to vocational education measures? Why or 
why not? 

Is [state] trying to develop or improve its student data system 
for general and academic education? What efforts are under 
way? How do these relate to vocational education data 
collection? Why or why not? Can you identify special 
populations in your current data system? 

Are there new programs to provide services to handicapped 
students, disadvantaged students or students with limited 
English proficiency? What is the nature of these programs? 
How do they relate to the requirements for service to special 
populations in vocational education? Why or why not? 

(Possibilities: general and vocational reforms are 
contradictory, are not related at all, are using similar 
approaches but remain separate, are integrated into a single 
system) 

Do you think vocational education accountability will play a greater 
or lesser role in genial education reform in the future? Why? 

Implementation of measures and standards 

What steps have been taken to implement performance measures 
and standards in [state] from the time they were adopted until the 
present? 

What have state agencies done in the areas of: 

- Dissemination of information about the measures and 
standards? 

— - Staff Development and Technical Assistance? 

— Data Collection? 

— Reporting? 

— Monitoring state and local actions? 

Which agencies/personnel have been involved? 
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What actions have they taken? 

How do these activities differ from the activities they were 
doing in the past? 

What level of resources is being devoted to the implementation of 
standards and measures? (e.g., How many staff are involved?) 

What have local programs done in these same areas? How does this 
differ from what they did in the past? 

What are the major problems that have been encountered in 
implementing measures and standards? 

How have attitudes toward measures and standards changed as the 
system has been developed and implemented? Improved? Grown 
more hostile? 

Technical issues regarding measures and standards 

What is the state doing to assess the reliability and fairness of the 
measures? 

Do you have any specific information about the quality of the 
measures? 

How important is comparability of data between programs? 

Will your system provide comparable data at the program 
level? How do you know? 

What is the state doing to assess the validity of the standards? 
How were standards set? 

Do you plan to review or revise the standards in the future? 

Do you permit local modifications to standards? Why or why not? If 
so, how well are they working? 

What is being done to encourage service to special populations? 

Are you using incentives or adjustments? Why or why not? 
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Use of measures and standards 

How are measures and standards supposed to be used at the state 
level (e.g., for monitoring, accountability, funding)? 

How will they be reported, and to whom? 

What use will be made of the information? 

How are measures and standards supposed to be used at the local 
level (e.g., for student counseling, program improvement)? 

How will data be reported, and to whom? 

What use will be made of the information? 

What is the state doing to affect the local use of this 
information? 

Are people concerned about the possible abuse of the measures and 
standards (e.g., creating incentives for programs to admit only the 
most qualified students)? 

What problems do you foresee? Why or why not? 

Assistance and Support for Measures and Standards 

What kinds of technical support do state offices need to implement 
measures and standards (e.g., specialized training)? 

Have you tried to obtain outside support? From whom? 

What kinds of technical assistance is needed at the local level? 

What is the state doing to provide it? 

What support will be provided specifically to encourage the use of 
measures and standards for local program improvement? 

What is the state's role in local program improvement? 
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Effects of measures and standards 

What effects have the measures and standards had so far (e.g., 
shifting responsibility from the state to the local level)? 

— on state agencies? 

— on local programs? 

— on the relationship between state agencies and local 
programs? 

What are the advantages and disadvantages of standards from the 
point of view of: 

— your agency? 

— other state agencies? 

— local programs? 

— business and industry? 
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RAND/MPR Interview Guide 
Visits to Local Districts 



Notes 



Respondents have been classified into categories, and sections of the 
interview are assigned to specific types of respondents: 

• Superintendent— includes state ; district and college chief ad- 
ministrators, e.g., chief state school officer, superintendent, 
chancellor, president. 

• Principal— includes local school chief administrators, e.g., prin- 
cipal, assistant principal. 

• Vocational Director— includes district or college vocational 
administrator, e.g., director of vocational education, vocational 
coordinator. 

• Teacher— includes college instructors and secondary teachers. 

• Other— includes administrators responsible for placement, spe- 
cial education, evaluation, data systems, etc. 



The general term "institution" is used in the interview guide. 
Substitute "state," "district," "college" or "school" as appropriate to 
the particular interview. 
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RAND/MPR Interview Guide 
Visits to Local Districts 

PERFORMANCE MEASURES AND STANDARDS 
[SUPERINTENDENT, PRINCIPAL] 

Adjust questions for standards and measures as implemented in 
each state, e.g., centralized vs. decentralized. 



Are you familiar with your state's 
performance measures and 
standards? (Ask by measure category) 

• Basic and advanced academic 
skills 

• Occupational competency 

• Job or work skill attainment 

• Retention 

• Placement 

• Other 

(decentralized] Which specific 
measures and standards has your 
institution adopted? 



When was the first time you were 
introduced to these measures and 
standards? 

Were you involved in their 

development? 

Has your institution added 

measures and standards to the 

state's set? 



How did you decide what to 
adopt? 

Who participated in the process? 
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THE NATURE AND USE OF STATEWIDE MEASURES AND 
STANDARDS [SUPERINTENDENT, PRINCIPAL] 



For what purposes will measures and 
standards be used in your institution 
(local leveM 

(e.g., for student counseling, program 
improvement) 


• How will data be reported, and to 1 
whom? 

• How will the information be used? 

• Will the information be used to 
compare programs? 


How will the data you report be used 
at the state level? 

(e.g., for monitoring, accountability, 
funding) 


• How will the data be reported, and 
to whom? 

• How will the information be used? 

• Will the information be used to 
compare programs or institutions? 

• How has the state communicated 
its intentions to you? 


Are people in this institution con- 
cerned about ths possible abuse of 
measures and standards? 
(e.g., creating incentives for 
programs to admit only the 
most qualified students) 


• Who has expressed concern? Why? 

• What problems do you foresee? 

• Why or why not? 
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IMPORTANCE OF MEASURES AND STANDARDS VIS-A-VIS 
OTHER VOCATIONAL EDUCATION INITIATIVES 
[VOCATIONAL DIRECTOR] 



Which vocational education reforms 
or initiatives have received the most 
emphasis in your institution in the 
past two years? 




How important are measures and 
standards in your institution relative 
to 

— other Perkins requirements, such 
as Tech Prep, integration of aca- 
demic and vocational education, 
special populations, etc.? 

— other state initiated vocational 
efforts? 

— district initiated vocational ef- 
forts? 

— school site initiated vocational 
efforts? 


• What activities have occurred that 
justify your judgment? 

• Can you give specific examples? 

• Which of these reforms will be 
more lasting? 

• Why do you think this is so? 


How do measures and standards 
relate to the local program applica- 
tion process? 

How do they fit in to the annual 
program evaluation process? 
How do the federal requirements 
differ from preexisting state policies 
or procedures? 


• What do you do differently now 
than you did prior to measures 
and standards? 

• How have reporting requirements 
changed under Perkins I and 
Perkins il? 


Are measures and standards sup- 
portive of or antagonistic to other 
vocational education reform efforts? 


• Which other vocational reforms? 

• How are they complementary or 
how do they conflict? 
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INTEGRATION OF VOCATIONAL EDUCATION 
ACCOUNTABILITY WITH OTHER EDUCATIONAL REFORMS 
[SUPERINTENDENT, PRINCIPAL] 



In education more broadly, which 
recent reform activities have received 

thit* mnct attpntinn in vour 

U1C lllU&l GLLC1 1 LAUl 1 Ul J \J\J~l 

institution? 


• How do these initiatives relate to 
accountability in vocational 
education? 


Is accountability an important aspect 
of general education reform? 


• What is being done to promote 
accountability? 

• How have vocational education 
measures and standards affected 
or been affected by this process? 

• Why or why not? 


Would the institution have taken 
actions to create or improve an 
accountability system without the 
federal requirement? 


• What evidence is there thot the 
institution was moving in this 
direction on its own? 

• What changes in assessment would 
have occurred? 

• What changes in data collection 
and use would have occurred? 


Is reform of assessment an important 
item in general education? In 
vocational education? 


• What assessment reforms are 
occurring? 

• How do they relate to vocational 
education measures? 

• Why or why not? 


Is the institution trying to develop or 
improve its student data system for 
general education? 


• What efforts are under way? 

• How do these efforts relate to 
vocational education data 
collection? Why or why not? 

• How are special populations 
identified? 

• How are tech prep students 
identified? 

• How do you define a "vocational 
student?" 


Are there new programs to provide 
services to handicapped students, 
disadvantaged students or students 
with limited English proficiency? 


• WMat is the nature of these 
programs? 

• How do they relate to the 
requirements for service to special 
populations in vocational 
education? 

• Why or why not? 
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Do you think voca tional education 
accountability will play a greater or 
lesser role in general education 
reform in the future? 
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IMPLEMENTATION OF MEASURES AND STANDARDS 
[SUPERINTENDENT, VOCATIONAL DIRECTOR, PRINCIPAL] 



Do the measures and standards 
apply to individual programs or to 
vocational education as a whole? 

Which local programs will be 
evaluated based on measures and 
standards? 


• Did you choose to concentrate 
Perkins funds on a limited number 
of sites or on a limited number of 
vocational programs? 

• HnpQ thp airnimtahilttv svstpm 

only apply to the projects, 
services, and activities assisted by 
Perkins funds? Or will measures 
and standards be applied more 
broadly to other programs? 

• Who is going to be required to 
report data? 


What steps have been taken to 
implement performance measures 
and standards from the time they 
were adopted until the present? 


• Which agencies or personnel have 
been involved? 

• What actions were taken? 

• Has your district added measures 
and standards to those adopted by 
the state? 


Who will be responsible for 
implementing the performance 

ITlcaoUf co dJIU olaJlUdiUo: 


• What responsibilities does this 
include? 

• Who else will be involved in the 
process? 

• What level of resources is being 
devoted to implementation? 


How much discretion do you have in 
how you implement the measures 
and standards? 


• What changes have you made, or 
could you make? 

• What supplemental requirement 
have you imposed? 


What are the major problems that 
have been encountered in 
implementing measures and 
standards? 


• What local conditions have been a 
barrier to implementation? 

• What has changed in the area of 
assessment and data collection? 

• How reasonable is the timeline 
requi^d by the state? 
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What is being done to encourage 
service to special populations? 


• Does the system include 
incentives or adjustments to 
encourage service? 

• How do your programs monitor 
and insure access, retention, and 
completion of special 
populations? 


How are performance measures and 
standards linked with the annual 
program evaluation? 

How are performance measures and 
standards involved in the program 
application process? 

Do performance measures and 
standards play a role in other Perkins 
requirements? 


• What is the connection between 
performance measures and 
standards and the annual evalua- 
tion? 

• What is the connection between 
performance measures and 
standards and the yearly appli- 
cation process for Perkins funds? 

• What other new Perkins data 
requirements have come from the 
state? 

• What new special population data 
requirements are there? 

• What are you doing to identify 
barriers to access or success and to 
evaluate the progress of special 
populations? 

• What are you doing to monitor 
teaching "all aspects of the 
industry?" 


Under what circumstances will you 
be required to write a local 
improvement plan ? 


• What directions has the state 
given you for developing a local 
improvement plan? 

• Who is responsible for 
determining improvement? 

• What is meant in this state by 
"substantial progress" toward 
achieving standards? 

• Who decided on the definition of 
this term? 
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Effects of Measures and Standards [Superintendent, 
Principal, Vocational Director] 



What changes have occurred in 
program evaluation procedures as a 
result of measures and standards? 

— in staff evaluation procedures? 

— in student evaluation procedures? 


How are programs/staff/students 
evaluated? 

How has this process changed in the 
past two years? 

What caused these changes? (Perkins 
or other factors) 

How will it change this year or next? 

[collect examples of relevant 
documents] 


What changes have occurred in data 
collection or data reporting? 

What changes are anticipated in the 

IUlUici 


Which data do teachers and 

administrators use to judge the 

quality of programs? 

Have there been changes in the 

available data during the past two 

years? 

If so, which data are no longer used, 

which are newly available? 

Will there be changes this year or 

next? 

[collect examples of relevant reports] 


What changes have occurred in staff 
responsibilities as a result of 
measures and standards? 

What changes are anticipated in the 
future? 


How many staff are involved with 
measures and standards? 
What do they do? 

What additional roles or 
responsibilities will evolve in the 
future? 


Have measu res and standards 
created any additional burdens for 
administrators or staff? 
Any other problems? 


What other demands do measures 

and standards make? 

What other problems have occurred? 


Do you have a Technical Advisory 
Group or something similar? 
How have measures and standards 
affected the role of the Technical 
Advisory Group? 


• How often does it meet? 

• Who participates? 

• Have there been any changes in 
the nature or function of the 
group? Why? 
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How have attitudes toward measures 
and standards chanced as the svstem 
has been implemented? 


• What were your first reactions? 

Othpr npnnlp'^ rpartinn*;? 

• How have those changed? 

• Can you give examples? 


Have you had to write any program 
improvement plans? 


• What criteria did you use to 
identify sites requiring program 
improvement plans? 

• What actions did the plans 
include? 

[collect examples of relevant 
documents] 


What effects have the measures and 
standards had so far on (e.g., 
shifting responsibility from the 
state to the local level): 

— state appncips? 

— local programs? 

— the relationship between state 
agencies and local programs? 




What are the advantages and 
disadvantages of measures and 
standards from the point of view of: 

— district 7 

— local programs? 

— business and industry? 




What do you anticipate the effects 
will be on program planning and 
instruction? 


• What kinds of changes might you 
make based on the data? 

• What kind of training or support 
would you need to use the data 
effectively? 

[collect examples of relevant 
documents] 


What effects will performance 
measures and standards have on 
special populations? 


• What steps have been taken to 
monitor access, progress, or 
success of special populations in 
quality vocational programs? 

• Are you applying the same 
standards to all programs, or 
separate standards for special 
populations? 

• Have the special populations 
served under Perkins changed 
over the past two years? 
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TECHNICAL ASSISTANCE AND SUPPORT FOR MEASURES 
AND STANDARDS [VOCATIONAL DIRECTOR] 



What technical assistance has been 
provided by the state? by the district? 




What kinds of technical support do 
districts/schools/institutions need to 
implement measures and standards? 
(e.g., specialized training) 


• Have you tried to obtain outside 
support? 

• From whom? 

• What is the state doing to provide 
it? 

• How is the state receiving the 
information on what technical 
assistance/support the local level 
needs? 


What are the most effective vehicles 
for technical assistance at the local 
level? 

(e.g., guidebooks, workshops, 
conferences, electronic mail, etc.) 


• Has the state held workshops? the 
district? 

• Have you attended AVA or other 
national conferences? 

• Do you receive information from 
NCRVE? 


What support will be provided 
specifically to encourage the use of 
measures and standards for locai 
program improvement? 


• What is the state's role in local 
program improvement? the 
district's role? the school's role? 
the teacher's role? 
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"GRASS-ROOTS" ISSUES [TEACHER] 



• How do you evaluate your program? 

* Do you have a Technical Advisory 
Group or something similar? 


• What kinds of data are collected? 

• How are the data used to judge the 
quality of programs and 
instructional strategies? 

• Is this a burden on you or your 
program? 

• What are you doing this year that is 
different than two years ago? 

• Does it play a role in program 
evaluation? 

• If so, how often does it meet and 
who participates? 

• How long has the group been 
around? 

• Have there been any changes in the 
nature or function of the group? 
Why? 


Are you familiar with your state's 
performance measures and 
standards? (Ask by measure cat- 
egory) 

• Basic and advanced academic skills 

• Occupational competency 

• Job or work skill attainment 

• Retention 

• Placement 

• Other 


• When was the first time you were 
introduced to these measures and 
standards? 

• Were you involved in their 
development? 


How have performance measures and 
standards affected you or your 
program? 

What do you anticipate the effects will 
be on program planning and 
instruction in the future? 


• Have they changed the way you 
evaluate the program? If so, how? 

• Have they changed the data you 
collect, report or use? If so, how? 

• Have they changed staff 
responsibilities? If so, in what 
ways? 

• Have they created additional 
burdens or other problems? If so, 
please describe. 
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How useful will performance 
measures and standards be for 
measuring the effectiveness of your 
program? 


• Will measures and standards help 
you improve your program? In 
what way? 

• Will measures and standards be 
used to compare your program to 
other programs? 

If yes, how do you feel about that? 


What assistance do you think you 
need in order to make the data useful? 


• What assistance have you received 
From whom? 

• Has u 2 state neiu worKbnopisi me 
district? 

• What kind of technical assistance is 
most helpful to you? (e.g., 
guidebooks, workshops, 
conferences, electronic mail, etc.) 

• Have you attended AVA or other 
national conferences? 

• Do you receive information from 
NCRVE? 


What does your program do to 
encourage service to special 
populations? 


• How do you monitor and ensure 
access, retention, and completion 
of special populations? 

• Have the types of special 
populations you serve changed over 
the past two years? 
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